Over-promise and under-deliver

Yep, that’s right. The title says it all. I promised to have an update available in mid-July. Now it’s August. Where is the update? Well, I have made some progress but it’s not ready yet. Sorry.

If you’re using hfcd, please download and overwrite the original hfcd.jar. Please use the following URL:

http://www.stopcoding.org/downloads/hfcd.jar

The expiration date is now October 1st, 2008.

Configuring Flex Builder to use the HellFire compiler

Hi, In my previous post, I talked about a RPC version of the Flex compiler API, i.e. instead of invoking the Flex compiler in the same JVM process as the caller, the RPC version makes network calls to a server dedicated to compiling Flex applications, modules and libraries. After working on the prototype a little more in the past several weeks, I finally have something for you to try - I call it the HellFire Compiler (yes, I give my work a name because ‘a RPC version of the Flex compiler’ is a mouthful).

There are three pieces in the HellFire Compiler:

1. A special version of flex-compiler-oem.jar. It is simply an implementation of the Flex Compiler API that makes network calls to a compiler daemon.
2. hfcd (HellFire Compiler Daemon). Users run hfcd as an independent, separate process (as a background job or in a Terminal window). hfcd uses the original flex-compiler-oem.jar to compile Flex applications.
3. An Eclipse/FB plugin. This is usually installed in FB. It retrieves from hfcd information about the compilations, e.g. compile time, link time, memory usage, etc.

Installation

So how do you go about configuring Flex Builder 3 to use the HellFire Compiler? First, we take care of hfcd…

1. Download and unzip hfcd_3.0.0.477.zip.
2. Download flex_sdk_3.0.0.477.zip and (do not unzip) and put it in the hfcd_3.0.0.477 directory.
3. Run ‘ant’ in the hfcd_3.0.0.477 directory.
4. You can now run hfcd in server/bin.

Please check out the screenshot below. It shows the expected results after step 3 and 4.

Now, we configure FB3 to use the RPC version of flex-compiler-oem.jar.

1. Run ‘Adobe Flex Builder 3′. Click ‘Preferences –> Flex –> Installed Flex SDKs’.

2. Click ‘Add’.

3. Click ‘Browse…’ and select ‘client’ in the hfcd_3.0.0.477 directory.

4. You should see ‘HellFire Compiler’ as the Flex SDK name. Click ‘OK’.

5. Select ‘HellFire Compiler’ as the default SDK. You can choose not to do this. But each Flex project must be individually configured to use the ‘HellFire Compiler’ sdk.

6. Click ‘OK’. Flex Builder 3 now uses the HellFire compiler to compile your opened projects. You should make sure hfcd is running before clicking ‘OK’.

Now, let’s install the Eclipse/FB plugin.

1. Click ‘Help –> Software Updates –> Find and Install…’.

2. Click ‘Search for new features to install’.

3. Click ‘New Local Site…’ and select ’site’ in the hfcd_3.0.0.477 directory.

4. The local site name is ‘hfcd_3.0.0.477/site’. You may change it to whatever you want.

5. Click ‘Finish’. Eclipse now reads site.xml in order to find out what are available.

6. Select ‘HellFire Compiler 3.0.0.477′ and click ‘Next’.

7. Read and accept the license agreement.

8. You may change the installation directory. I would not. Click ‘Finish’.

9. The plugin is not digitally signed, but it’s okay. Click ‘Install All’.

10. Click ‘Yes’ when you are asked to restart Flex Builder.

11. You should now see the ‘HellFire Compiler’ view in FB.

12. Click ‘Project –> Clean’ and start a build immediately. You will see in the ‘HellFire Compiler’ view what hfcd has compiled and the applications/libraries compile/link time.

The software will expire on August 1st, 2008. But I’ll likely have an update available before the expiration.

Note that this software has not reached production quality yet. If you see any issues, please email hfcd@stopcoding.org

Uninstallation

1. Run ‘Adobe Flex Builder 3′. Click ‘Preferences –> Flex –> Installed Flex SDKs’.

2. Select ‘HellFire Compiler’. Click ‘remove’ and ‘OK’.

3. Click ‘Help –> Software Updates –> Manage Configuration’.

4. Right-click ‘HellFire Compiler’. Select ‘Uninstall’. Restart Flex Builder.

What’s next?

There are several areas I would like to explore

• With hfcd, the compiler and the other FB features are separated and it’s possible to give the VM for hfcd specific VM args that boost the performance of the compiler. For example, I’ve seen consistent 30% performance improvement when hfcd uses the ‘-server’ VM mode. It may be true or maybe I forgot to blink my eyes. I don’t know. Also, it’s interesting to see how aggressive the VM grows the heap in ‘-server’ VM mode.
• Running hfcd on a second machine. When I was with Adobe, there was a desktop under my desk (I don’t know why I put a *desktop* computer *under* my desk) that I occasionally used for testing purposes. It was faster than my laptop too! So in theory, if you run hfcd on such a machine, it could shorten the FB build time…. I’m going to see if Eclipse can help me synchronize files and remap paths….
• Suppose that there are two independent projects in your workspace, i.e. no project references. In theory, Eclipse/FB could compile the projects in parallel. If so, it would make the out-of-process solution even more compelling.
• The Eclipse/FB plugin currently only retrieves compile time and link time. But there is nothing to prevent the plugin from retrieving compiler reports (e.g. dependency info, usage info), bytecode disassembler output, etc.

The Flex Compiler API… RPC Style!

I don’t know how many of you had used Flex Builder 1.0… It wasn’t Eclipse-based. It was based on the award-winning Dreamweaver C++ codebase. We all know that the compiler has been Java-based since day one, so it’s somewhat difficult to integrate the compiler with the C++ based FB (what JNI??). Anyhow, if I remember correctly (and Heidi can correct me if I’m wrong), FB1 launches the mxmlc executable out-of-process in order to compile. There are some drawbacks with this approach. Obviously, the first one is the JVM launch time. Also, compiler error messages go to stderr - FB1 has to parse the messages in stderr. To make matter worse, there are subtle formatting differences in the error messages that makes parsing and tabulating them in FB1 somewhat challenging.

So in Flex 2, the compiler team was asked to address those 1.0 problems. Back then, it was a consensus among the compiler folks that it would be nice to be able to programmatically call the compiler in-process. That should solve the two main problems I described above.

My first attempt for FB2 was what I called the low-level compiler API (check out the classes in the flex2.compiler package and the main() method in mxmlc). You could pick and choose the ‘compilers’ and ‘transcoders’ and other pieces based on the hosting application’s requirements. Well, it sounds great. However, it’s too flexible - FB2, the hosting application, needs to have a lot of new codes in order to integrate with the low-level API. So after the release of FB2 and with the feedback from the FB team, I built on top of the low-level compiler API a high-level Flex compiler API (the flex2.tools.oem package) for FB2.0.1 and FB3. With this new compiler API, one can programmatically compile a Flex application with a few lines of Java codes:

Application app = new Application(new File(”helloworld.mxml”));
Configuration c = app.getDefaultConfiguration();
c.setSourcePath(…);
app.setConfiguration(c);
app.setOutput(new File(”helloworld.swf”));
app.build(true)

I can talk about how to use the Flex Compiler API here but Matt Horn’s Flex Compiler Guide (PDF) is the best IMO! Please check it out.

What I really want to talk about here is this in-process vs. out-of-process debate.

Well, I actually don’t want to debate because it is evident that FB moves from one set of (out-of-process) problems in version 1 to a new set of (in-process) problems in version 2 and 3. The most obvious one is that the compiler competes with the other features in FB3 (e.g. smart editing, profiling, design view) for CPU and memory resources.

So, is it a bad decision to move from doing out-of-process to in-process? My answer is no. Some customers have only a couple of projects in their workspace so the in-process approach works just fine. However, some of them have over 20+ so the out-of-process approach (I mean out-of-process and on another machine) provides a viable option. In other words, you need FB to give you both options.

But FB3 is out and it only supports in-process compilations and you can’t wait for the next release? Well, the good news is, FB3 uses the Flex Compiler API. So, all you need is implement a RPC version of the Flex Compiler API; add out-of-process to FB3 as a new SDK; run a server that uses the original flex-compiler-oem.jar and viola…

Since I designed and implemented the API and know the API very well, it’s pretty easy for me to implement the RPC version. In fact, I spent the last few days doing a prototype and IT WORKS! Click the thumbnail image below to see the screenshot.

I hope to polish it up and make it available as soon as possible. If you’re willing to be an early alpha tester, please shoot me an email.

The Future of The Flex Profiler

If you haven’t read my previous blog about the Flex Profiler, I recommend you go take a look…

Ever since the Flex Profiler was made available in early FB3 beta, I’ve been hearing some legitimate concerns. One example and I paraphrase:

It’s great that Flex has implemented a large number of Eclipse views for the main use cases. But they are not showing me what I want…

Another example:

The steps of showing profiling data are quite simple and easy to remember. But once the data is shown, it seems to me that it usually takes too many steps to navigate to the root cause of the performance/memory problems. Why can’t the tool be smarter?

Like I said, these are legitimate concerns and I’m sure the Flex engineering team is going to improve the profiler going forward. How? I’m going to give my $0.02 here..

Just to recap, there are 4 pieces in the profiler architecture:

1. The flash.sampler.* API support and the PreloadSwf support in the Player
2. ProfilerAgent.swf as the PreloadSwf
3. Java-based profiler client library
4. Eclipse plugin + the graphical user interface

As you can see, if the current Flex profiler doesn’t work for you and you can’t wait for any improvements in the next release, your most logical option will be the flash.sampler API, i.e. write your own profiler agent or simply use the flash.sampler classes in your application. There are many advantages and disadvantages.

Advantages:

• Free. You don’t have to pay several hundred dollars more for FB Pro. Well, your time is money too. so, this point doesn’t make sense.
• Your profiler agent could be very lightweight if you know exactly what you’re looking for.

Disadvantages:

• You want a profiler agent that is lightweight, i.e. not using up too much of CPU cycles and Player memory. However, it could take you a long time to write a good one.
• You have to use another timing approach to verify the correctness of your implementation.
• You may waste your valuable time if you have to move your sampling code from one wrong spot to another in your application.

I think most developers would avoid writing their own. At some point, some developers/software vendors may make theirs available (free of charge? who knows?). I don’t see that in my crystal ball yet.

If the current Flex profiler works for you but you want some enhancements, feel free to submit your enhancement requests. However, they may not work on every enhancement (e.g. they may not build 15 more new profiler views). Instead, you should tell Adobe the following:

• Expose the API of the Java-based profiler client library. The API returns pre-calculated data that’s ready for tabulation. It can also return raw data from ProfilerAgent.swf if you want to do your own calculations.
• Expose the API of the foundation classes (e.g. the base class of the performance/memory views) and the utility classes.
• Define extension points in the Eclipse-based profiler plugin.

If Adobe listens to you and do the above, it will be very easy for you to extend the Flex profiler. I think it’s in their best interest to provide the flexibility you need because the Player doesn’t provide profiling support in a closed environment - the flash.sampler API is available to everyone. Sooner or later, the Flex Profiler will not be the only game in town.

Another great thing that developers can do if the API of the profiler client library is exposed is static code analysis. Did you notice that in the inclusion/exclusion filter dialog, it automatically shows a list of packages as you type?

The package list is built dynamically by the Flex Profiler by decoding the bytecodes in your application. Yep, the application SWF is available via the profiler client library API. That’s why I think someone could write a profiler extension plugin that uses SWFKit (the open source SWF encoding/decoding library) to decode and run static code analysis. It sure would be very nice to be able to do both static/dynamic code analysis with the Flex Profiler.

Let’s Talk About The Flex Profiler

Yeah, I know. This is a compiler blog, not a profiler blog. But I can explain…

The performance and memory profiling feature in Flex Builder Professional Edition wasn’t done by the FB development team. It was done by the Flex compiler team. Yes. I mean everything: the Eclipse plugin, user interface, profiler agent, data collection and processing!

So now you know that the compiler team was not only busy with the compiler, but also the profiler, during the Flex 3 development cycle. And that explains why I’m “eligible” to talk about the profiler here…

Those who have been closely following Flex since version 1 should notice that profiling was present in version 1 and 1.5 but absent in version 2 and 2.0.1. Why no profiler in version 2? It’s a long story. But it was a combination of poor scheduling, coordination and… well… lack of technological advances.

You may ask, what technological advances? Well, if you had used Flex 1.0 and 1.5 in the past, you should know that there was a performance profiler (http://www.adobe.com/devnet/flex/articles/profiler.html). The performance profiler in Flex 1.x was trace-based, i.e. the Player records timing data for everything that happens in the old AVM. The upside is that it captures everything. The downside is that it makes the Player run slowly and the amount of data for a short period of time (say 10 seconds) could be huge and take time to process. I’m not suggesting that the Flex 1.x performance profiler is not useful. It needs you to know roughly where to optimize. In fact, thanks to the profiler, we were able to improve Flex 1.5 framework startup performance by over 30%. Yahoo! was able to speed up their Flash/Flex-based Yahoo! Maps before the debut. But the breakthrough came when Tom Reilly added support for sample-based profiling to the latest Player…

I’m making an assumption here that you all studied statistics when you were in college and know what sampling is. If not, ask the people around you about the purpose of national presidential polls you hear these days. Same idea. Of course, I’m very sure that Tom’s stuff for identifying performance bottlenecks in your code is a lot more accurate than those national polls for predicting the next president of the United States!

Unlike traced-based, with sample-based profiling, you don’t get everything. But your application runs just as fast (or as slow) when sampling is enabled. Because it’s sampling, you don’t get a notification every time your application enters or exits a function. But you still get a pretty accurate performance profile.

So far so good, but there are some negatives. For instance, it is pretty tricky to find, given a function that is getting called excessively, the time distribution from the list of callers of that function.

Other than performance profiling, the Player now also supports memory profiling. It reports object instantiations, object deletions and where objects were instantiated.

So far, I’ve talked quite a bit about the Player side of the Flex profiler story. Let’s turn to the profiler architecture. It is actually pretty simple. There are 4 pieces:

1. The flash.sampler.* API support and the PreloadSwf support in the Player
2. ProfilerAgent.swf as the PreloadSwf
3. Java-based profiler client library
4. Eclipse plugin + the graphical user interface

For performance profiling, the flash.sampler API returns sampling data as a stream of stack traces. So, it’s pretty easy to process. For memory profiling, the API returns object instantiation and deletion events and also stack traces indicating where objects were instantiated.

The PreloadSwf support is new in the latest Player. The idea is that, if you specify a local SWF in your mm.cfg, the Player will load/run the specified SWF in parallel to the target SWF. These two crucial Player features together provide the one-two punch we desperately needed against slow-running Flash movies.

Now anyone could write some AS3 codes to collect performance and memory data. For the Flex Builder profiling feature, we implemented a non-GUI-based AS3 program (ProfilerAgent.swf) that collects and sends performance/memory data to Flex Builder via socket connections.

On the client side, there are two pieces. The first piece is the Eclipse plugin and the graphical user interface. The “Flex Profiling” perspective and the associated views (a total of 11) are a major addition to Flex Builder 3. The user interface is aimed at:

1. identifying performance bottlenecks
2. exposing excessive object allocations
3. finding loitering objects

The second piece is the profiler client library. Obviously, it’s for maintaining server socket connections; taking performance/memory data; processing them and archiving them. We need this library because we need a demarcation between the Eclipse code and the non-Eclipse code so that QA can use the Eclipse-free client library to write automated test cases for ProfilerAgent.swf.

If you want to know how to use the Flex Profiler, please check out the livedocs (http://livedocs.adobe.com/flex/3/html/help.html?content=profiler_3.html).

To Be Continued…

I Don’t See These Tricks In fcsh!

In one of my previous posts, Flex Compiler SHell (fcsh), I mentioned that Flex removed the flex2.tools.Shell class from mxmlc.jar in version 3. For those who’ve read my previous post, flex2.tools.SimpleShell is a simplified version of flex2.tools.Shell and is currently the ‘official’ fcsh. But flex2.tools.Shell is the version I used back then when I worked for Adobe. This does a lot more and I believe you will find that useful. So let me give you a demo here.

I’m going to use the ‘checkinapp’ application in my demo. It is located at flex/sdk/branches/3.0.x/frameworks/tests/checkinapp.

Adobe Flex Compiler SHell (fcsh) Version 3.0.0
Copyright (c) 2006 Adobe Systems Incorporated. All rights reserved.

(fcsh) mxmlc -debug=true -optimize=false checkinapp.mxml
fcsh: Assigned 1 as the compile target id
Loading configuration file frameworks/flex-config.xml
checkinapp/checkinapp.swf (756154 bytes)

To see the dependencies of checkinapp.mxml, we use the ‘dependency’ command. The number ‘1′ is the compile target id. “Top-level Definitions” means the definitions are visible externally to the file. “Inheritance” means they are the superclass and implementing interfaces. “Namespace” means they are the AS3 namespaces found in the application. “Types” means the listed definitions are found in the API signatures. “Expressions” are the definitions that are found elsewhere in the file, e.g. method bodies.

(fcsh) dependency 1 checkinapp.mxml
checkinapp/checkinapp.mxml
===Top-level Definitions===
checkinapp
===Inheritance===
mx.binding:IBindingClient
mx.core:Application
flash.events:IEventDispatcher
===Namespace===
AS3
mx.core:mx_internal
===Types===
mx.events:SliderEvent
ShoppingCart
PanelControl
ThumbnailView
Checkout
mx.styles:CSSStyleDeclaration
mx.utils:ObjectProxy
mx.events:FlexEvent
mx.containers:Panel
flash.events:Event
GridView
mx.effects:Dissolve
mx.containers:Canvas
mx.binding:IWatcherSetupUtil
mx.controls:Image
mx.containers:HBox
ProductDetail
mx.effects:Fade
flash.events:MouseEvent
CartView
mx.controls:HSlider
mx.events:ListEvent
mx.core:UIComponentDescriptor
mx.effects:Resize
mx.binding:Binding
mx.containers:ViewStack
ResizePanelEvent
===Expressions===
mx.controls:LinkButton
_checkinappWatcherSetupUtil
mx.controls:Label
ShoppingCart
mx.controls:Button
mx.events:PropertyChangeEvent
mx.controls:Spacer
PanelControl
ThumbnailView
mx.styles:StyleManager
Checkout
checkinapp_thumbRoll
mx.styles:CSSStyleDeclaration
mx.utils:ObjectProxy
mx.containers:Panel
checkinapp_listRoll
GridView
checkinapp_thumbOff
mx.containers:Canvas
flash.net:navigateToURL
mx.controls:ComboBox
mx.controls:Image
checkinapp_listOn
flash.utils:getDefinitionByName
checkinapp__embed_mxml_assets_products_tg_logo_gif_1172660324
mx.containers:HBox
ProductDetail
mx.containers:VBox
CartView
mx.controls:HSlider
mx.core:UIComponentDescriptor
mx.effects:Resize
mx.containers:ControlBar
checkinapp_listOff
flash.net:URLRequest
mx.binding:Binding
mx.containers:ViewStack
checkinapp_thumbOn

To see the usage of mx.controls:Button, we use the “usage” command. For example, ScrollThumb uses Button as a superclass.

(fcsh) usage 1 mx.controls:Button
[expressions] checkinapp/checkinapp.mxml
[types] framework.swc(mx/controls/scrollClasses/ScrollBar)
[inheritance] framework.swc(mx/controls/scrollClasses/ScrollThumb)
[expressions] checkinapp/ProductDetail.mxml
[expressions] checkinapp/Checkout.mxml
[types] framework.swc(mx/containers/Panel)
[types] checkinapp/CartView.mxml
[types] framework.swc(mx/controls/NumericStepper)
[inheritance] framework.swc(mx/controls/sliderClasses/SliderThumb)
[inheritance] framework.swc(mx/controls/CheckBox)
[types] framework.swc(mx/containers/Accordion)
[types] framework.swc(mx/controls/ComboBase)
[inheritance] framework.swc(mx/controls/RadioButton)
[inheritance] framework.swc(mx/containers/accordionClasses/AccordionHeader)
[types] framework.swc(mx/controls/alertClasses/AlertForm)
[inheritance] framework.swc(mx/controls/LinkButton)
Found: 16

I think many of you have seen and tried the “-link-report” mxmlc command-line option. Personally, I don’t use it because the -link-report XML output could easily reach .5MB in size for even the smallest possible Flex applications. Also, fcsh is better because the “usage-dependency” combo allows you to traverse the dependency graph in either directions.

To see the number of ‘debuggable” lines, we use the “lines” command. The counts below exclude blank lines. Also, for MXML files, the counts do not count XML tag lines, only the lines in <mx:Script>. The most interesting number below, IMO, is the size of the autogenerated code - the percentage is pretty high. The autogenerated codes are mainly for MXML components (e.g. data binding, styles, etc). That tells you one thing: if you were to write those MXML components in AS3, not in MXML, you would have to write a whole lot more. On the other hand, of course, one could argue that the compiler should autogen as little as possible.

(fcsh) lines 1
188 checkinapp/CartView.mxml
697 checkinapp/Checkout.mxml
26 checkinapp/CreditCardPicker.mxml
87 checkinapp/GridView.mxml
129 checkinapp/PanelControl.mxml
191 checkinapp/ProductDetail.mxml
151 checkinapp/ProductThumbnail.mxml
8 checkinapp/ResizePanelEvent.as
27 checkinapp/ShoppingCart.as
325 checkinapp/StatePicker.mxml
106 checkinapp/ThumbnailView.mxml
649 checkinapp/checkinapp.mxml
total number of lines in the source files: 2584
total number of lines in the autogen files: 1424

To see the bytecode size (in bytes) of the definitions, we use the “sizes” command. I’m listing only the top ten classes here. But you know there are hundred of classes in checkinapp.

(fcsh) sizes 1
104300 framework.swc(mx/controls/listClasses/ListBase)
75913 framework.swc(mx/core/UIComponent)
62331 framework.swc(mx/controls/DataGrid)
47073 framework.swc(mx/core/Container)
39207 framework.swc(mx/controls/dataGridClasses/DataGridBase)
39010 framework.swc(mx/controls/listClasses/TileBase)
34284 framework.swc(mx/controls/List)
30588 framework.swc(mx/managers/SystemManager)
28816 checkinapp/checkinapp.mxml
27623 framework.swc(mx/controls/Button)

I think these fcsh commands should help you analyze your codes’ dependencies and sizes and should in turn, minimize your application’s size.

SWC: More Should Be Done For Tooling

If you are a Flex developer, I am sure that you know what SWC is. SWC files are component archive files (a.k.a. libraries) and it’s the library format of choice for Flash Authoring and Flex. SWCs are zip-compatible - that means you can use any zip utilities to open and view it.

The SWC format was co-developed by Flash Authoring and Flex several years ago - right around the period when the Flex project started. In a typical SWC, you can find the following files:

• library.swf
• catalog.xml
• some PNG images

library.swf contains component bytecodes and assets in the SWF file format. This allows the compilers in Flash Authoring and Flex to pull the bytecodes out of the SWC and insert it as-is into the target SWF. Flash Authoring can also use library.swf in the ‘preview’ section.

catalog.xml describes the list of components and classes and in the SWC file and also states the dependencies of the classes in the file. The dependency information in catalog.xml is important for the compiler and linker because they provide information to the compiler what classes to pull out and to the linker how the classes to be lined up in the target SWF. Some may ask why need catalog.xml when library.swf has the classes? Well, the classes are stored in library.swf linearly, not in a dependency graph format. That’s why library.swf won’t help the linker much.

SWC files may also contains asset files. For example, PNG images can be placed in a SWC so that Flash Authoring can use the PNG images in the component panel. Fast forward to the Flex 3 SDK release, you can see that we expand the concept of asset files to CSS files and resource bundles (.properties files).

Now, let’s take a step back and look at SWC again. The general idea of SWCs is that they are replacements of source files for tools when source files are not available. It is a compact, binary or semi-binary format. But as of its current form, it is ‘compact’ because it does not hold all of the information that you would normally find from source files. In other words, SWC does not retain enough information for tooling.

If you were about to build a tool that would support both source files and SWCs, you would need two separate codepaths for all the features in your tool. If you follow this line of thinking, you can easily see what are missing in SWC. I can list a couple here:

1. character offsets for all the method signatures (e.g. method name, input parameter name and type, return type) and variable definitions in a source file.
2. API comments

There may be more than that for tooling. Please speak out if you see more tooling use cases. I know the above two because I saw them first-hand in FB and asdoc.

My understanding is that compc knows everything but it does not split out everything it knows into the SWC, so my $0.02 is that they are not hard to solve.

Flex Compiler SHell (fcsh)

In early summer 2005, the Flex development team was busy working on Flex 2. The new Flex compiler redesign and implementation were finally gaining traction (well, you have no idea how frustrating it was when you had to build a new compiler that integrated with some unstable AS3 compiler pieces and had to run those resulting SWFs in an unstable VM). Anyhow, Dan Smith and I believed that it was the right time to discuss with the Flex Builder team about integrating the Flex compiler into FB2.

FB2 is an Eclipse-based IDE for Flex application development. Similar to other IDEs, FB2 should be able to compile Flex applications incrementally. Back then, the new compiler could barely generate good SWFs and the compiler’s full compile performance was not something we were proud of. But you know, things will get better over time, so I decided to switch focus and work a little bit on the incremental compilation support.

Now that the Flex compiler is open source, you should be able to trace how incremental compilation is done. But it might be difficult to identify the code that is responsible for incremental compilation - there is nothing like incrementalCompile() or something like that. One reason is that the compiler algorithms (API.batch1() and API.batch2()) are actually designed for incremental compilations. They treat full compilations as special cases, i.e. a full compilation is a case where everything needs to be recompiled.

Frankly, it wasn’t that hard to nail the basic incremental compilation support as long as you were dealing one unified algorithm for full vs. incremental. But FB at that point did not have any code in it that would invoke the compiler. Well, for testing, I had to write a tool (not just some test cases) to prove that incremental compilation worked.

To simulate how the compiler works in an IDE, what could be better than doing it in some sort of shell environment (Well, writing an Eclipse plugin as a simulator is straightforward too, but that’s not the point!). Yup, a shell. That’s what I built - Flex Compiler SHell (fcsh). Note that it’s SHell, not Shell. Some official Flex documentation are still using ‘Shell’ - I don’t like that.

It took me a couple of days to get fcsh up and running. There are so many upsides using fcsh than mxmlc and/or compc. One of them is that you eliminate JVM startup time and those 2000+ compiler classes that JVM needs to load for even typical Flex applications. Those overhead could add up to 2 seconds. Another advantage is of course, what it’s built for - incremental compilation. I wasted no time dumping mxmlc and compc and used fcsh exclusively for my daily development work.

Soon after its availability, other folks on the team found good use of it. The QA and doc team dramatically cut their time for building their applications by using fcsh to batch-process their apps.

Due to the nature of my work, i.e. compiler development, I needed to be able to get to data that I couldn’t find in a SWF, e.g. dependency info, bytecode sizes, etc. For this reason, I added a slew of fcsh commands:

• abc - outputs abc bytecodes given a definition name
• dependency - displays a list of dependencies given a file name
• files - lists all the source and SWC files that contribute to the resulting SWF
• sizes - lists the bytecode size of all the definitions in the resulting SWF
• lines - displays the number of ‘debuggable’ lines in the resulting SWF
• memory - displays memory usage (when using JDK 1.5)
• mm.cfg - displays settings in mm.cfg
• trace - turns tracing on/off in mm.cfg
• exportorder - displays the export order of the definitions in the resulting SWF

At first, product management and engineering management weren’t warm to the idea of adding a new tool to the SDK without getting requests from our customers. That’s why you don’t see fcsh in the Flex 2 SDK release.

But the Flex product management reversed their decision and shipped fcsh in Flex 2.0.1. That was great news. But the bad news was that the above-mentioned fcsh commands must be removed.

If you still have Flex SDK 2.0.1 installed, you can check out fcsh.jar. I think it points to flex2.tools.SimpleShell. If you change the manifest to point to flex2.tools.Shell, you will get the full version.

If you only have Flex SDK 3 installed, you’re out of luck because flex2.tools.Shell is no longer distributed. In my opinion, the Flex product/engineering management should consider adding the flex2.tools.Shell class back into the distribution.

Understanding the Flex Compiler

STOP! Before you continue, I want to make sure that you have downloaded the compiler source code from the Adobe open source site. Ideally, it would really help if you also have Eclipse setup so you can view the source files easily.

As I mentioned in my previous post, the Flex compiler supports multiple programming languages. In the Flex compiler, these languages are compiled by a family of language-specific compilers. In Eclipse –> Open Type, just type “Compiler” (see below), you should see the compiler classes for the supported languages.

Obviously one can not assume that all programming languages can be compiled in one step. For example, a MXML component may use components written in AS3. Before compiling that MXML component down to some bytecodes, the compiler must locate the dependent AS3 components and verify the methods that the MXML component invokes for correctness.

It is also reasonable to assume that they may not be compiled in the same number of steps. For example, MXML needs twice the number of steps than AS3.

In the Flex compiler, all the language-specific compilers implement the same compiler contract, which is the flex2.compiler.Compiler interface (see below).

As you can see, other than the isSupported() and getSupportedMimeTypes() methods, which are available for the top-level Flex compiler to know which language the Compiler instance supports, the compilation is a 9-step process.

The top-level Flex compiler acts as a coordinator. It is responsible for invoking the Compiler instances. The Compiler instances can expect the call sequence to look something like this:

1. preprocess (once)
2. parse1 (once)
3. parse2 (once)
4. analyze1 (once)
5. analyze2 (once)
6. analyze3 (once)
7. analyze4 (once)
8. generate (once)
9. postprocess (multiple times until the Compiler instance requests a stop)

When the top-level Flex compiler is not calling these methods, it does a number of things (but not necessarily limited to):

• picks the appropriate Compiler instance based on the source file type;
• learns about unresolved types from the Compiler instances and searches for them in the source-path and library-path;
• passes type information to the Compiler instances;
• decides which Compiler instance should proceed based on the states of the Compiler instances and the overall resource allocation situation

In order for the top-level Flex compiler to run the show, it requires the Compiler instances to cooperate. Basically, it requires them to produce certain type of information. For example,

• A syntax tree must be available at the end of parse2().
• analyze1() must identify the superclass name.
• analyze2() must identify the remaining dependencies.
• analyze4() must make the fully-resolved type info available.

The top-level Flex compiler continues to run the compilation process until:

• It no longer needs to look for new dependencies.
• The Compiler instances report errors.

As I mentioned above, the top-level Flex compiler would invoke those 9 Compiler methods. Although the call sequence is well-defined, the top-level Flex compiler can still have many different ways to get the compilation done. In fact, there are two compiler algorithms in the Flex compiler. One (flex2.compiler.API.batch1()) is structured and another one (flex2.compiler.API.batch2()) is opportunistic.

API.batch1() is a conservative and more structured algorithm. The main characteristic of this algorithm is that it makes sure that the compilation of all the source files reaches the same state before it proceeds to the next state, e.g. analyze1() gets called for all the files before analyze2().

API.batch2() is a more opportunistic algorithm. The goal of this algorithm is to minimize memory usage. Unlike API.batch1(), source files with fewer dependencies could reach generate() well before source files with more dependencies reach analyze3(). The idea is that as long as a source file gets compiled down to some bytecodes, compiler resources allocated to that file (e.g. memory for the syntax tree) can be freed up immediately.

Now, you pretty much know the basics of the Flex compiler infrastructure. Let’s recap:

1. The top-level Flex compiler uses either batch1() or batch2() compiler algorithm to compile.
2. The compiler algorithms use two different strategies to invoke those 9 methods in the Compiler instances.
3. The participating Compiler instances must cooperate by providing certain information to the top-level Flex compiler at the end of each Compiler method.
4. The top-level Flex compiler infrastructure is responsible for the dirty works like source-path/library-path searching, hooking up loggers for error reporting, etc. The Compiler implementations do not have to worry about that.

One thing that’s worth noting is that the command-line tools (e.g. mxmlc, compc, asdoc) and the Flex Compiler API all use the same above-mentioned infrastructure and algorithms.

I can talk a little more about this topic but I will stop here and let you read some codes. I will continue in the next post but if you want me to talk about other topics, please drop me a line. Thanks.

A Brief History of The Adobe Flex Compiler

Good evening. Let me start with telling a little bit about the history of the Flex compiler. When we first started building the Flex compiler back in 2002, we almost started everything from scratch. Well, one of the exceptions was the support for ActionScript 2. We ported the C++ version of the ActionScript 2 compiler written by the Flash Authoring team to Java. We then rolled our own Java-based SWF encoding/decoding library (back then we called it SWFKit… now I don’t know what is officially called) and then added a whole bunch of features like MXML, SWC and asset imports. It was quite an experience because we “glue” this version 1.0 compiler together from a bunch of codes we produced and grabbed from Authoring and other open source projects. The “borrowed” codes didn’t have well-defined integration points. Some of them needed refactorings. The resulting 1.x compiler was okay - we were proud of that but we all believed back then we could make it better in our second attempt.

That opportunity came in 2004. We were wrapping up the Flex 1.5 development but we were also busy planning on what to do in Flex 2. At that time, we knew we were going to have a brand-new VM and “compiler” for ActionScript 3 support (The Player team was in charge of building both). We were excited but worried - we needed to focus on wrapping up 1.5 but we also needed to set aside engineering time to review the new AS3 compiler design. Basically, we couldn’t afford to get a “black-box” AS3 compiler again. We needed the AS3 compiler to come in pieces with well-defined integration points. We needed the AS3 compiler pieces to fit right into our new Flex 2.0 compiler architecture.

Fortunately, we got what we wanted… but not without spending a couple of months to port the C++based AS3 compiler into Java and test the Java port against some several thousands SpiderMonkey tests! Well, this was hard work but without intimately involved in the porting work, we would not have been able to understand the design and recommend design changes.

We also rewrote and refactored most of the MXML parser and AS code generation. SWC was also redesigned and reimplemented to address scalability issues.

The compiler codebase that is now open source, although in version 3, is still based on the compiler architecture we designed in version 2. In version 3, we added features like cross-domain RSL support; static multi-language compilation; Adobe Font Engine support, etc.

IMHO, this is the result of the hard work done by the compiler team. Frankly, it is pretty amazing to see that a compiler can support multiple languages/text formats (.mxml, .as, .css, .properties) and do all these by seamlessly tracking dependencies automatically.

The story is obviously much longer than the above, but I want to spare you from continuing to read this boring stuff. My next post will talk about how to understand the compiler step-by-step.