If Scala runs on the JVM, how can Scala do things that Java seemingly cannot?

Question

I just learned about Scala yesterday, and I'd like to learn more about it. One thing that came to mind, however, from reading the Scala website is that if Scala runs on the JVM, then how is it possible for bytecode compiled from Scala source to be able to achieve things that Java can't readily do, such as (but not limited to) reified generics?

I understand that the compiler is what generates the bytecode; so as long as the compiler can massage source code into valid bytecode supported by the JVM, then both should be equivalent. But I was under the impression that Java couldn't even reify its own generics, so how could another compiler be able to pull this off?

Answer 1

"All" programming languages run on x86, so how can they be much different from each other?

Brainfuck and Haskell are both Turing complete, so they can both do the exact same tasks.

There's a bit of room for syntax changes, syntax sugar and compiler magic in between. You can do quite a lot in there, but there is always a limit. In your case, it's JVM byte code.

If Java can produce the same byte code as Scala, they are equivalent. It could, however, be the case that a new feature in the JVM gets implemented in only Scala. Very unlikely, but possible.

Answer 2

To address the specific issue that you raise, of reified generics . . .

In many contexts, type parameters are actually saved in class-files and exploitable via reflection, even despite erasure. For example, the following program prints class java.lang.String:

import java.lang.reflect.Field;
import java.lang.reflect.ParameterizedType;
import java.util.ArrayList;

public class ErasureWhatErasure {
    private final ArrayList<String> foo = null;

    public static void main(final String... args) throws Exception {
        final Field fooField = ErasureWhatErasure.class.getDeclaredField("foo");
        final ParameterizedType fooFieldType =
            (ParameterizedType) fooField.getGenericType();
        System.out.println(fooFieldType.getActualTypeArguments()[0]);
    }
}

All "erasure" means is that when you create an instance of a parameterized type, the type parameter is not recorded as part of the instance; new ArrayList<String>() and new ArrayList<Integer>() create identical instances. But even in Java, there are a few well-known workarounds, such as:

1. Something like new ArrayList<String>() { } actually creates a new instance of an anonymous subclass of ArrayList<String>, and the subclassing relationship does record the type parameter. So you can retrieve the String reflectively. (Specifically: ((ParameterizedType) new ArrayList<String>() { }.getClass().getGenericSuperclass()).getActualTypeArguments()[0] is String.class.) This is exploited by various frameworks, such as Guice and Gson.
2. If you're creating your own class, you can require the type parameter to be passed via the constructor, and store it in a field. (The compiler can help you enforce that the constructor argument matches the type parameter. And if the type argument is itself generic, you can combine this with workaround #1 to ensure that you have the whole type argument.)

Another language residing on the JVM could reify generics by making #2 happen implicitly; whenever you declared a generic class, the type-parameter field(s) and constructor-parameter(s) would be added implicitly, and whenever you instantiate or extend it, type-argument(s) would be implicitly copied to constructor-argument(s) to populate them. Java already does the same thing — implicit fields and constructor-parameters/arguments — in a different context, namely, for local classes that refer to final local variables in their containing methods.

The main limitation is that generic classes in the JDK — the collections framework, java.lang.Class, etc. — don't already have this setup, and alternative languages running in the JVM can't add it to them. So such languages would need to provide their own equivalents of the JDK classes. But they can still use the JVM itself.

Answer 3

JVM bytecode pretends to be a kind of generic machine code, and it is indeed, so... what makes you think it couldn't support any other language? JVM bytecode is a Turing complete language, and thus, every program, no matter in which language is written, can be compiled/translated into bytecode.

There are a lot of languages which already have a bytecode compiler (e.g. Jython for Python and JRuby for Ruby) and there are also quite different from Java.

Note that technically, every Turing complete programming language can be compiled into another. It could be possible to compile JS to C or Ruby to Python, for example.

Answer 4

In a nutshell, Scala can do it because the Scala compiler is a master in code transformation/generation. Which means Java could do it too (maybe it does already). The trick is not made at the bytecode level but at the source level.

Could you guess the output of this code:

def test[T](f : => Any) : T = {
  try { val x = f.asInstanceOf[T]
    println("f.asInstanceOf did not raise an error")
      x
  }
  catch { case e : Throwable =>
    println("f.asInstanceOf did raise an error")
    throw e
  }
}

val x = test[Int]("x")

(Not so) Suprisingly it outputs

f.asInstanceOf did not raise an error
java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:105)

... 33 elided

Because of type erasure, f.asInstanceOf[T] can not fail but obviouly a String is not an Int so it has to fail somewhere. Fortantely Scala brings solutions to this problem:

import shapeless.Typeable
import shapeless.syntax.typeable._

def test[T : Typeable](f : => Any) : T = {

    f.cast[T] match {
        case Some(x) => {
            println("f.cast[T] succeed")
            x
        }
        case None    => {
            println("f.cast[T] failed")
            throw new RuntimeException("cast failed!")
        }
    }
}

val x = test[Int]("x")

The main difference here is we declare T as Typeable. Scala have several ways to provide runtime representations of types.

Answer 5

Reified generics do not require JVM support. Yes, they would be easier and most performant with JVM support, but JVM support is not necessary. For instance, a Scala compiler could, for every class featuring a type variable, add a field that stores the corresponding object:

class List<T> {

}

void test() {
    List<?> list = new List<String>();
    List<Integer> intList = (List<Integer>) list;
}

would be compiled to

class List<T> {
    final Class<T> tClass;
    public List(Class<T> tClass) {
        this.tClass = tClass;
    }

    public <O> List<O> castTo(Class<O> oClass) {
        if (tClass == oClass) {
            return (List<O>) this;
        } else {
            throw new ClassCastException("Incompatible type parameter: " + tClass);
        }
    }
}

void test() {
    List<?> list = new List<String>(String.class);
    List<Integer> intList = list.castTo(Integer.class);
}

There are of course more elaborate translation strategies that more seamlessly integrate with the host type system, for instance by actually having separate classes for different type arguments to the same generic type:

class List<T> {

}

class List#Integer extends List<Integer> { }

class List#String extends List<String> { }

void test() {
    List<?> list = new List#String();
    List<Integer> intList = (List#Integer) list;
}

(There would of course be some challenges not apparent in this trivial example, for instance recursive type arguments, proper isolation of different class loaders, ...)

The reason Java does not have reified generics is not that reification would be impossible on the JVM, but that reified generics would have broken backwards compatibility (in particular binary compatibility) with existing Java programs - something Scala did not have to worry about.