–tools-locale in Flex 4.x.x is very useful, but…

There is a new compiler argument in Flex 4.x. It’s –tools-locale. For example, specifying –tools-locale=en would instruct the compiler to output errors/warnings in English. The following is the output of “mxmlc -help tools-locale”:

-tools-locale <string>
specifies the locale used by the compiler when reporting errors and warnings.

Of course, the compiler, by default, uses Locale.getDefault() (i.e. most of the time, JVM -Duser.language). Having this compiler argument is useful for tool integration because a tool could use the OS default language but the compiler could output English errors/warnings.

But why wouldn’t developers choose to read compiler errors/warnings in their native languages and would like to have a way to configure the compiler to output in e.g. English? Is it because the translations are not good? Can someone please comment on the qualities of the non-English compiler error/warning messages?

The Flex Compiler API… RPC Style!

I don’t know how many of you had used Flex Builder 1.0… It wasn’t Eclipse-based. It was based on the award-winning Dreamweaver C++ codebase. We all know that the compiler has been Java-based since day one, so it’s somewhat difficult to integrate the compiler with the C++ based FB (what JNI??). Anyhow, if I remember correctly (and Heidi can correct me if I’m wrong), FB1 launches the mxmlc executable out-of-process in order to compile. There are some drawbacks with this approach. Obviously, the first one is the JVM launch time. Also, compiler error messages go to stderr – FB1 has to parse the messages in stderr. To make matter worse, there are subtle formatting differences in the error messages that makes parsing and tabulating them in FB1 somewhat challenging.

So in Flex 2, the compiler team was asked to address those 1.0 problems. Back then, it was a consensus among the compiler folks that it would be nice to be able to programmatically call the compiler in-process. That should solve the two main problems I described above.

My first attempt for FB2 was what I called the low-level compiler API (check out the classes in the flex2.compiler package and the main() method in mxmlc). You could pick and choose the ‘compilers’ and ‘transcoders’ and other pieces based on the hosting application’s requirements. Well, it sounds great. However, it’s too flexible – FB2, the hosting application, needs to have a lot of new codes in order to integrate with the low-level API. So after the release of FB2 and with the feedback from the FB team, I built on top of the low-level compiler API a high-level Flex compiler API (the flex2.tools.oem package) for FB2.0.1 and FB3. With this new compiler API, one can programmatically compile a Flex application with a few lines of Java codes:

Application app = new Application(new File(“helloworld.mxml”));
Configuration c = app.getDefaultConfiguration();
c.setSourcePath(…);
app.setConfiguration(c);
app.setOutput(new File(“helloworld.swf”));
app.build(true)

I can talk about how to use the Flex Compiler API here but Matt Horn’s Flex Compiler Guide (PDF) is the best IMO! Please check it out.

What I really want to talk about here is this in-process vs. out-of-process debate.

Well, I actually don’t want to debate because it is evident that FB moves from one set of (out-of-process) problems in version 1 to a new set of (in-process) problems in version 2 and 3. The most obvious one is that the compiler competes with the other features in FB3 (e.g. smart editing, profiling, design view) for CPU and memory resources.

So, is it a bad decision to move from doing out-of-process to in-process? My answer is no. Some customers have only a couple of projects in their workspace so the in-process approach works just fine. However, some of them have over 20+ so the out-of-process approach (I mean out-of-process and on another machine) provides a viable option. In other words, you need FB to give you both options.

But FB3 is out and it only supports in-process compilations and you can’t wait for the next release? Well, the good news is, FB3 uses the Flex Compiler API. So, all you need is implement a RPC version of the Flex Compiler API; add out-of-process to FB3 as a new SDK; run a server that uses the original flex-compiler-oem.jar and viola…

Since I designed and implemented the API and know the API very well, it’s pretty easy for me to implement the RPC version. In fact, I spent the last few days doing a prototype and IT WORKS! Click the thumbnail image below to see the screenshot.

I hope to polish it up and make it available as soon as possible. If you’re willing to be an early alpha tester, please shoot me an email.

Flex Compiler SHell (fcsh)

In early summer 2005, the Flex development team was busy working on Flex 2. The new Flex compiler redesign and implementation were finally gaining traction (well, you have no idea how frustrating it was when you had to build a new compiler that integrated with some unstable AS3 compiler pieces and had to run those resulting SWFs in an unstable VM). Anyhow, Dan Smith and I believed that it was the right time to discuss with the Flex Builder team about integrating the Flex compiler into FB2.

FB2 is an Eclipse-based IDE for Flex application development. Similar to other IDEs, FB2 should be able to compile Flex applications incrementally. Back then, the new compiler could barely generate good SWFs and the compiler’s full compile performance was not something we were proud of. But you know, things will get better over time, so I decided to switch focus and work a little bit on the incremental compilation support.

Now that the Flex compiler is open source, you should be able to trace how incremental compilation is done. But it might be difficult to identify the code that is responsible for incremental compilation – there is nothing like incrementalCompile() or something like that. One reason is that the compiler algorithms (API.batch1() and API.batch2()) are actually designed for incremental compilations. They treat full compilations as special cases, i.e. a full compilation is a case where everything needs to be recompiled.

Frankly, it wasn’t that hard to nail the basic incremental compilation support as long as you were dealing one unified algorithm for full vs. incremental. But FB at that point did not have any code in it that would invoke the compiler. Well, for testing, I had to write a tool (not just some test cases) to prove that incremental compilation worked.

To simulate how the compiler works in an IDE, what could be better than doing it in some sort of shell environment (Well, writing an Eclipse plugin as a simulator is straightforward too, but that’s not the point!). Yup, a shell. That’s what I built – Flex Compiler SHell (fcsh). Note that it’s SHell, not Shell. Some official Flex documentation are still using ‘Shell’ – I don’t like that. 😛

It took me a couple of days to get fcsh up and running. There are so many upsides using fcsh than mxmlc and/or compc. One of them is that you eliminate JVM startup time and those 2000+ compiler classes that JVM needs to load for even typical Flex applications. Those overhead could add up to 2 seconds. Another advantage is of course, what it’s built for – incremental compilation. I wasted no time dumping mxmlc and compc and used fcsh exclusively for my daily development work.

Soon after its availability, other folks on the team found good use of it. The QA and doc team dramatically cut their time for building their applications by using fcsh to batch-process their apps.

Due to the nature of my work, i.e. compiler development, I needed to be able to get to data that I couldn’t find in a SWF, e.g. dependency info, bytecode sizes, etc. For this reason, I added a slew of fcsh commands:

• abc – outputs abc bytecodes given a definition name
• dependency – displays a list of dependencies given a file name
• files – lists all the source and SWC files that contribute to the resulting SWF
• sizes – lists the bytecode size of all the definitions in the resulting SWF
• lines – displays the number of ‘debuggable’ lines in the resulting SWF
• memory – displays memory usage (when using JDK 1.5)
• mm.cfg – displays settings in mm.cfg
• trace – turns tracing on/off in mm.cfg
• exportorder – displays the export order of the definitions in the resulting SWF

At first, product management and engineering management weren’t warm to the idea of adding a new tool to the SDK without getting requests from our customers. That’s why you don’t see fcsh in the Flex 2 SDK release.

But the Flex product management reversed their decision and shipped fcsh in Flex 2.0.1. That was great news. But the bad news was that the above-mentioned fcsh commands must be removed.

If you still have Flex SDK 2.0.1 installed, you can check out fcsh.jar. I think it points to flex2.tools.SimpleShell. If you change the manifest to point to flex2.tools.Shell, you will get the full version.

If you only have Flex SDK 3 installed, you’re out of luck because flex2.tools.Shell is no longer distributed. In my opinion, the Flex product/engineering management should consider adding the flex2.tools.Shell class back into the distribution.

Understanding the Flex Compiler

STOP! Before you continue, I want to make sure that you have downloaded the compiler source code from the Adobe open source site. Ideally, it would really help if you also have Eclipse setup so you can view the source files easily.

As I mentioned in my previous post, the Flex compiler supports multiple programming languages. In the Flex compiler, these languages are compiled by a family of language-specific compilers. In Eclipse –> Open Type, just type “Compiler” (see below), you should see the compiler classes for the supported languages.

Obviously one can not assume that all programming languages can be compiled in one step. For example, a MXML component may use components written in AS3. Before compiling that MXML component down to some bytecodes, the compiler must locate the dependent AS3 components and verify the methods that the MXML component invokes for correctness.

It is also reasonable to assume that they may not be compiled in the same number of steps. For example, MXML needs twice the number of steps than AS3.

In the Flex compiler, all the language-specific compilers implement the same compiler contract, which is the flex2.compiler.Compiler interface (see below).

As you can see, other than the isSupported() and getSupportedMimeTypes() methods, which are available for the top-level Flex compiler to know which language the Compiler instance supports, the compilation is a 9-step process.

The top-level Flex compiler acts as a coordinator. It is responsible for invoking the Compiler instances. The Compiler instances can expect the call sequence to look something like this:

1. preprocess (once)
2. parse1 (once)
3. parse2 (once)
4. analyze1 (once)
5. analyze2 (once)
6. analyze3 (once)
7. analyze4 (once)
8. generate (once)
9. postprocess (multiple times until the Compiler instance requests a stop)

When the top-level Flex compiler is not calling these methods, it does a number of things (but not necessarily limited to):

• picks the appropriate Compiler instance based on the source file type;
• learns about unresolved types from the Compiler instances and searches for them in the source-path and library-path;
• passes type information to the Compiler instances;
• decides which Compiler instance should proceed based on the states of the Compiler instances and the overall resource allocation situation

In order for the top-level Flex compiler to run the show, it requires the Compiler instances to cooperate. Basically, it requires them to produce certain type of information. For example,

• A syntax tree must be available at the end of parse2().
• analyze1() must identify the superclass name.
• analyze2() must identify the remaining dependencies.
• analyze4() must make the fully-resolved type info available.

The top-level Flex compiler continues to run the compilation process until:

• It no longer needs to look for new dependencies.
• The Compiler instances report errors.

As I mentioned above, the top-level Flex compiler would invoke those 9 Compiler methods. Although the call sequence is well-defined, the top-level Flex compiler can still have many different ways to get the compilation done. In fact, there are two compiler algorithms in the Flex compiler. One (flex2.compiler.API.batch1()) is structured and another one (flex2.compiler.API.batch2()) is opportunistic.

API.batch1() is a conservative and more structured algorithm. The main characteristic of this algorithm is that it makes sure that the compilation of all the source files reaches the same state before it proceeds to the next state, e.g. analyze1() gets called for all the files before analyze2().

API.batch2() is a more opportunistic algorithm. The goal of this algorithm is to minimize memory usage. Unlike API.batch1(), source files with fewer dependencies could reach generate() well before source files with more dependencies reach analyze3(). The idea is that as long as a source file gets compiled down to some bytecodes, compiler resources allocated to that file (e.g. memory for the syntax tree) can be freed up immediately.

Now, you pretty much know the basics of the Flex compiler infrastructure. Let’s recap:

1. The top-level Flex compiler uses either batch1() or batch2() compiler algorithm to compile.
2. The compiler algorithms use two different strategies to invoke those 9 methods in the Compiler instances.
3. The participating Compiler instances must cooperate by providing certain information to the top-level Flex compiler at the end of each Compiler method.
4. The top-level Flex compiler infrastructure is responsible for the dirty works like source-path/library-path searching, hooking up loggers for error reporting, etc. The Compiler implementations do not have to worry about that.

One thing that’s worth noting is that the command-line tools (e.g. mxmlc, compc, asdoc) and the Flex Compiler API all use the same above-mentioned infrastructure and algorithms.

I can talk a little more about this topic but I will stop here and let you read some codes. I will continue in the next post but if you want me to talk about other topics, please drop me a line. Thanks.