Jeff Sinclair’s Website

There is often confusion around what the permanent generation contains and given its name, whether or not it can be garbage collected. To cut a long story short, the permanent generation can be garbage collected and is where reflective class meta data, as well as string constants are stored.

This blog entry takes a practical approach to answer the following questions:

• Can the permanent generation be garbage collected?
• Does the value of -Xmx (the maximum heap size) include the permanent generation?

Can the permanent generation be garbage collected?

Yes it can. There is often the misconception that string constants cannot be garbage collected. However the only requirement that the JVM specification stipulates is that identity comparison works for constant string values. There is no requirement that the constant be the same object throughout the lifetime of the JVM. Hence if there are no references left to a string constant, there is no reason why it cannot be garbage collected.

To prove this take the following code:

 
public class PermGenDemo {
  public static void main(String[] args) {
    int i = 0;
    while (true) {
      ("string-" + ++i).intern();
    }
  }
}

If the above code is run using the following command:

 
$ java -verbose:gc -XX:PermSize=8m -XX:MaxPermSize=64m PermGenDemo

The following output will be seen (note output cut for brevity):

 
[Full GC [PSYoungGen: 32K->0K(28928K)] . . . [PSPermGen: 65535K->6461K(65536K)]]
[Full GC [PSYoungGen: 32K->0K(31168K)] . . . [PSPermGen: 65535K->6461K(65536K)]]

The above conclusively shows that the permanent generation is collected and when it requires collecting, triggers a full collection.

Does the value of -Xmx (the maximum heap size) include the permanent generation?

No it does not. The size of the permanent generation is controlled by the following JVM options:

* -XX:MaxPermSize controls the maximum permanent generation size.
* -XX:PermSize controls the initial permanent generation size.

The permanent generation is like any other generational compartment: it can grow to a maximum, technically can shrink and can also be garbage collected.

To demonstrate that the permanent generation is not included in the maximum heap settings, pmap can be used (e.g. by running pmap <pid> | tail -1).

The following table illustrates the output of pmap using different JVM heap and permanent generation settings:

In the above -Xms=-Xmx and -XX:PermSize=-XX:MaxPermSize are set deliberately to stop the heap expanding and ensure constant results. The first value of 512204K gives us a base value which takes into account any overhead introduced by native libraries (including the jvm itself) being mapped into memory. The second value of 760556K is an exact increase of 256m, which is exactly how much the heap was increased by. The third value of 825188K is an exact increase of 64m, which is exactly how much the permanent generation was increased by.

The above results clearly show that the settings for permgen are unrelated to that of the heap.

Conclusion

• The permanent generation can be garbage collected like other generations.
• The permanent generation is additional memory to the java heap.
• -XX:MaxPermSize controls the maximum memory used by the permanent generation.
• -XX:PermSize controls the initial memory used by the permanent generation.

Resources

1. Java Hotspot VM Options
2. Presenting the Permanent Generation

java hotspot, java, jvm

Several well known JVM tools appear to be able to magically detect the Java processes running on a system. Good examples of this are: jps and Visual VM. As of Java 6 it is even possible to find out yourself which Java processes are running on the local machine that you have access to using the Dynamic Attach API. These APIs can be found in the tools.jar (JAVA_HOME/lib/tools.jar).

The following code illustrates how the Dynamic Attach API can be used to list all of the local virtual machine identifiers (which for the HotSpot VM map onto the JVM PID):

 
package uk.co.cooljeff.dynamicattach;
import java.util.List;
import com.sun.tools.attach.VirtualMachine;
import com.sun.tools.attach.VirtualMachineDescriptor;
public class JVMFinder {
  public static void main(String[] args) {
    List<VirtualMachineDescriptor> vmDescriptors = VirtualMachine.list();
    for (VirtualMachineDescriptor vmDescriptor : vmDescriptors) {
      System.out.println("Name: " + vmDescriptor.displayName()
                                                    + " PID: " + vmDescriptor.id());
    }
  }
}

The above gives the following output on my machine (cut out a few for brevity):

 
Name: uk.co.cooljeff.dynamicattach.JVMFinder PID: 21116

I’m about to spoil the fun now for those of you who were hoping to see some wacky OS dependent mechanism or hidden RMI registry for finding this information. The solution is quite simple, the JVM dumps a file containing performance data to a standard location which it uses to work out what JVMs are running on a host.

Specifically the performance data is for jvmstat, however it is this file that forms the basis of the Dynamic Attach implementation that HotSpot uses.

The following piece of code replicates what the LocalVmManager (part of jvmstat) does to locate running Java processes:

 
package uk.co.cooljeff.dynamicattach;
import java.io.File;
import java.io.FilenameFilter;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import sun.jvmstat.perfdata.monitor.protocol.local.PerfDataFile;
public class JVMDataLocation {
  public static void main(String[] args) throws Exception {
    final Pattern filePattern = Pattern.compile(PerfDataFile.userDirNamePattern);
    final Matcher fileMatcher = filePattern.matcher("");
    FilenameFilter fileFilter = new FilenameFilter() {
      public boolean accept(File dir, String name) {
        fileMatcher.reset(name);
        return fileMatcher.matches();
      }
    };
    File[] files = new File(PerfDataFile.getTempDirectory()).listFiles(fileFilter);
    for (File file : files) {
      System.out.println("PerfDataDir: " + file);
    }
  }
}

Which gives the following on my machine:

 
PerfDataDir: /tmp/hsperfdata_root
PerfDataDir: /tmp/hsperfdata_tomcat55
PerfDataDir: /tmp/hsperfdata_jeffsinc

If you take a look at one of these directories you will find 1 file per Java process running by the specific user. The name of the file is the local virtual machine identifier (lvmid) which for HotSpot will correspond to the process id. If you run strings on the file you will see all of the properties relating to that JVM instance.

It is as simple as that, on Linux the JVM simply looks for all files that match: /tmp/hsperfdata_*/*.

Resources:

• Dynamic Attach API
• OpenJDK 7 source code
• Visual VM

java hotspot, java, jvm

When I asked a colleague at work what I should do during my vacation, he said build the OpenJDK… so I did!

In hindsight, it would have been easier to have built IcedTea (RedHat’s OpenJDK hybrid) rather than Sun’s raw OpenJDK. However since I spent quite a bit of time setting up the environment I was determined to get it to build.

At a high level I did the following:

• Downloaded the source (~ 85Mb): http://download.java.net/openjdk/jdk7/.
• Downloaded the binary plugins (the bits of the OpenJDK that are yet to be open sourced).
• Downloaded the dependencies.
• Modified some environment variables.
• Checked the environment is ok via the sanity script.
• Made 1 change to the source code.
• Made 1 change to the make compiler flags.
• Ran the build.

The README file did not cover all of the dependencies needed to build the OpenJDK on Ubuntu, here is what worked for me:

• sudo apt-get install build-essential
• sudo apt-get install gawk
• sudo apt-get install m4
• sudo apt-get install libasound2-dev
• sudo apt-get install libcupsys2-dev
• sudo apt-get install sun-java6-jdk
• sudo apt-get install x11proto-print-dev
• sudo apt-get install libxp-dev
• sudo apt-get install libfreetype6-dev
• sudo apt-get install libxt-dev
• sudo apt-get install libxtst-dev

The OpenJDK comes with a script (jdk/make/jdk_generic_profile.sh) to help set up your environment. The main updates I needed to make were:

• The bootjdk variable which holds the folder name of your Java 6 install.
• The jdk_instances variable which holds the parent directory of your bootjdk.
• The binplugs variable which holds the location of the non open source bits of the OpenJDK.
• Ensure ANT_HOME was exported.
• Ensusre ANT_HOME/bin was on the PATH.

You can see my full environment here: jdk_generic_profile.

The environment was then verified using the sanity make target:

 
$ . jdk/make/jdk_generic_profile.sh
$ make sanity

Once this passed I tried building but got a couple of errors.

The first error was:

 
/openjdk/hotspot/src/share/vm/libadt/port.hpp:
In function 'void bcopy(const void*, void*, size_t)':
/openjdk/hotspot/src/share/vm/libadt/port.hpp:40:
error: 'void bcopy(const void*, void*, size_t)' redeclared inline
without 'gnu_inline' attribute
/usr/include/bits/string3.h:90:
error: 'void bcopy(const void*, void*, size_t)' previously defined here
make[6]: *** [incls/_precompiled.incl.gch] Error 1

I understand what the error means but I could not work out why the error occurs because after looking through /usr/include/string.h, the include of /usr/include/bits/string3.h is only done if the user specifies _FORTIFY_SOURCE which as far as I can tell is not done in the OpenJDK. My workaround is quite simple, I commented out the redeclaration of inline void bcopy(const void*, void*, size_t) since it appears to already be inlined somehow.

The second error was:

 
/openjdk/hotspot/src/share/vm/runtime/arguments.cpp:
In static member function 'static void Arguments::set_aggressive_opts_flags()':
/openjdk/hotspot/src/share/vm/runtime/arguments.cpp:1359:
error: format '%d' expects type 'int', but argument 3 has type 'intx'

This occurs because of a flag that treats WARNINGS as ERRORS. The correct approach would have been to go and fix the warnings, instead I disabled the flag WARNINGS_ARE_ERRORS = -Werror which can be found in hotspot/make/linux/makefiles/gcc.make.

After making these changes I was able to run the build (just typed make) and saw the following output in the end:

 
Control linux amd64 1.7.0-internal build_product_image build finished:
Control linux amd64 1.7.0-internal all_product_build build finished:
Control linux amd64 1.7.0-internal all build finished:

I had a working JDK which could be found in build/linux-amd64/j2sdk-image/bin.

 
$ ./java -version
openjdk version "1.7.0-internal"
OpenJDK Runtime Environment (build 1.7.0-internal-jeffsinc_2009_01_03_10_52-b00)
OpenJDK 64-Bit Server VM (build 14.0-b09, mixed mode)

java java, openjdk

APT has interested me for quite some time because after reading various blog entries and articles, it was unclear to me where APT fits in relation to other tools that provide functionality triggered by annotations. Specifically I’ve been interested in any overlap between APT and AspectJ.

Both AspectJ and APT provide the following common functionality:

• Ability to intercept annotations.
• Ability to declare custom compiler errors and warnings based off annotation matching.

I attempted to implement an architecture enforcement requirement using both tools. The requirement was simple: a compile error should occur when an attempt is made to access a class annotated with @RestrictedResource unless the caller is annotated with @SuppressRestrictions.

This was relatively simple with AspectJ, the following Aspect provides the enforcement:

 
public aspect RestrictedResourceEnforcer {
    private pointcut accessToRestrictedResource() :
        call(* (@RestrictedResource *).*(..))
                || call((@RestrictedResource *).new(..));
    private pointcut withinUnrestrictedCode() :
        @withincode(SuppressRestrictions) || @within(SuppressRestrictions);
    declare error : accessToRestrictedResource() && !withinUnrestrictedCode():
        "Attempt to access a restricted resource.";
}

The following test demonstrated the compiler error as well as the lack of compiler error should @SuppressRestrictions be used:

 
public class RestrictedResourceEnforcerTest {
    public void testAccessToRestrictedResource() {
        // Compile error: attempt to access restricted resource.
        RestrictedClass restrictedClass = new RestrictedClass();
        // Compile error: attempt to access restricted resource.
        restrictedClass.method1();
    }
    @SuppressRestrictions
    public void testAccessToSuppressedRestrictionResource() {
        RestrictedClass restrictedClass = new RestrictedClass();
        restrictedClass.method1();
    }
    @RestrictedResource
    private static class RestrictedClass {
        public void method1() {
        }
    }
}

Immediately I realised that it is not possible to implement this kind of architecture enforcement using APT without annotating the calling class (i.e RestrictedResourceEnforcerTest). Without marking the calling class, I’m not going to get a process() callback from APT.

This attempt has shown me that APT is only of real use for processing that is completely contained within the annotated class.

I still wanted try APT in direct comparison with AspectJ so I tried enforcing immutability on any class with an @Immutable annotation. I gave up pretty quickly using APT for this because although it is technically possible, it requires a lot of work with ASM to generate the correct source to enforce immutability (i.e. return unmodifiable collections on getters, throw an exception with setters). AspectJ provides a much more maintainable solution without ever having to go near something like ASM.

So I’m left wondering if APT has any value to your regular Java developer (rather than toolkit developer), I don’t think it has. APT’s use-case looks purely in the space of creating extensions to the regular Java compiler to generate bye products directly related to the class with the annotation.

As a general rule I’d break down APT and AspectJ usage as follows:

• Use AspectJ if you need to add a functional requirement to existing entities. Examples include: monitoring, architecture enforcement, transactional functionality.
• Use APT if you need to generate bye products for framework integration. Examples include: schema generation, source code generation tools (e.g. JAXB, JAXWS)

APT is not designed to be updating code to meet a functional requirement, AspectJ is. Instead APT provides a compiler extension that allows you to generate bye products driven by meta data on a class.

java apt, aspectj, java