–tools-locale in Flex 4.x.x is very useful, but…

There is a new compiler argument in Flex 4.x. It’s –tools-locale. For example, specifying –tools-locale=en would instruct the compiler to output errors/warnings in English. The following is the output of “mxmlc -help tools-locale”:

-tools-locale <string>
specifies the locale used by the compiler when reporting errors and warnings.

Of course, the compiler, by default, uses Locale.getDefault() (i.e. most of the time, JVM -Duser.language). Having this compiler argument is useful for tool integration because a tool could use the OS default language but the compiler could output English errors/warnings.

But why wouldn’t developers choose to read compiler errors/warnings in their native languages and would like to have a way to configure the compiler to output in e.g. English? Is it because the translations are not good? Can someone please comment on the qualities of the non-English compiler error/warning messages?

HFCD 4 Now Supports Flex and Flash Builder 4 Final!

Last week was an exciting week. Flex 4 and Flash Builder 4 were out FINALLY! I had HFCD 3 going final as well… actually, one business day before the Flex 4 official announcement!

Now that Flex 4 and Flash Builder 4 are available, it’s time to upgrade HFCD 4 from supporting Gumbo beta 2 to the Flex 4 final release. I spent some time last week porting and testing some codes and I’m pleased to announce that HFCD 4 (remains in beta) now supports Flex 4 and Flash Builder 4.

It’s true that HFCD 4 currently remains in beta. I just want to hear some more feedback and work out some more kinks before upgrading its status. If you ask me, I’d rate this update closer to release candidate than early beta w.r.t. quality.

As I mentioned in my previous post, HFCD is now a product of Bytecode Workshop, Inc. You can download the installers there. Several useful links are below:

• main site: http://bytecode-workshop.com
• installers: http://bytecode-workshop.com/#section=downloads
• documentation: http://bytecode-workshop.com/#section=docs
• support: http://bytecode-workshop.com/#section=support

After using the compiler in Flex 4 for a few days, I can see that the compiler is faster than its Flex 3 counterpart. Congrats to the folks at Adobe on this achievement. But since HFCD uses whatever is improved in Flex and it’s architected differently (and of course, with the use of multi-core technology), HFCD continues to outperform Flex in most medium-to-large project workspace situations. The build performance when combining Flash Builder 4, Flex 4 and HFCD 4 is truly amazing and I had already talked about it awhile back.

My plan is that HFCD 4 will not stay long in the beta status and will move aggressively towards the final release, so please help me by trying it when you get a chance. Thanks!

A New Version of HFCD

Recently I picked up the hfcd project again and finally added features that I promised last year. Yes, it’s a new version of hfcd. This morning, I’ve finished testing what I got so far. It’s in beta quality, but I think I should release it so you folks can give it a spin and give me some feedback.

Let me give a brief intro of what hfcd is. HellFire Compiler Daemon is a RPC-style Flex Compiler server. Clients use the Flex Compiler API to communicate with hfcd. Basically, client processes use hfcd to compile Flex applications. The most obvious client is Flex Builder 3. FB3, by using the hfcd client SDK, can offload compilations to hfcd so that its memory and CPU resources can be reserved for other FB3 popular tasks: design view, code model, profiling, just to name a few. The result is dramatic. hfcd compiles Flex applications much faster in most cases than compiling apps in-memory with FB3.

The hfcd version I released last year was an proof-of-concept attempt. One limitation was that it had to run locally with FB3 (i.e. on the same machine). Now, with the new features recently added to hfcd, it’s able to run on a remote machine. This implies that if the remote machine has more memory and faster processors, the build time could be dramatically reduced. This is certainly one option of improving the compiler performance.

Furthermore, I added the following features:

1. hfcd supports a virtual file system that holds source files in memory. It means your hfcd server machine will never be littered with codes from many different sources.
2. hfcd supports background compilations, both full and incremental. This is a game changer. Why? Developers usually edit multiple files before they build (Ctrl-B) and test. As long as developers save changes continuously (instead of keeping all the changes to the one-and-only Ctrl-S), the background compilation could kick in and compile some changes while developers are making the rest of the changes. The result, developers wait much shorter incremental build time after the last Ctrl-S and Ctrl-B.
3. hfcd takes advantage of multi-processor-core technology wherever possible. It attempts to build applications and libraries that have no mutual dependencies simultaneously. For example, if you have 2 SWC projects and 3 SWF projects. The SWC projects don’t depend on each other. With FB3, your build time would be: T(SWC1 + SWC2 + SWF1 + SWF2 + SWF3). With FB3 + hfcd, your build time would be roughly: T(max(SWC1, SWC2) + max(SWF1, SWF2, SWF3))! Another game changer!

The currently supported SDK version is 3.3.0.4852. I hope to add support for 3.4.0 in the near term. UPDATE (Nov 25th, 2009): Now HFCD supports Flex SDK 3.4.1 and 4.0.0.

In the next few weeks, I need some developers to test my first beta. If you are interested, please email hfcd@stopcoding.org. This weekend, I hope to get a web page ready so you folks can register and download there.

The Flex Compiler API… RPC Style!

I don’t know how many of you had used Flex Builder 1.0… It wasn’t Eclipse-based. It was based on the award-winning Dreamweaver C++ codebase. We all know that the compiler has been Java-based since day one, so it’s somewhat difficult to integrate the compiler with the C++ based FB (what JNI??). Anyhow, if I remember correctly (and Heidi can correct me if I’m wrong), FB1 launches the mxmlc executable out-of-process in order to compile. There are some drawbacks with this approach. Obviously, the first one is the JVM launch time. Also, compiler error messages go to stderr – FB1 has to parse the messages in stderr. To make matter worse, there are subtle formatting differences in the error messages that makes parsing and tabulating them in FB1 somewhat challenging.

So in Flex 2, the compiler team was asked to address those 1.0 problems. Back then, it was a consensus among the compiler folks that it would be nice to be able to programmatically call the compiler in-process. That should solve the two main problems I described above.

My first attempt for FB2 was what I called the low-level compiler API (check out the classes in the flex2.compiler package and the main() method in mxmlc). You could pick and choose the ‘compilers’ and ‘transcoders’ and other pieces based on the hosting application’s requirements. Well, it sounds great. However, it’s too flexible – FB2, the hosting application, needs to have a lot of new codes in order to integrate with the low-level API. So after the release of FB2 and with the feedback from the FB team, I built on top of the low-level compiler API a high-level Flex compiler API (the flex2.tools.oem package) for FB2.0.1 and FB3. With this new compiler API, one can programmatically compile a Flex application with a few lines of Java codes:

Application app = new Application(new File(“helloworld.mxml”));
Configuration c = app.getDefaultConfiguration();
c.setSourcePath(…);
app.setConfiguration(c);
app.setOutput(new File(“helloworld.swf”));
app.build(true)

I can talk about how to use the Flex Compiler API here but Matt Horn’s Flex Compiler Guide (PDF) is the best IMO! Please check it out.

What I really want to talk about here is this in-process vs. out-of-process debate.

Well, I actually don’t want to debate because it is evident that FB moves from one set of (out-of-process) problems in version 1 to a new set of (in-process) problems in version 2 and 3. The most obvious one is that the compiler competes with the other features in FB3 (e.g. smart editing, profiling, design view) for CPU and memory resources.

So, is it a bad decision to move from doing out-of-process to in-process? My answer is no. Some customers have only a couple of projects in their workspace so the in-process approach works just fine. However, some of them have over 20+ so the out-of-process approach (I mean out-of-process and on another machine) provides a viable option. In other words, you need FB to give you both options.

But FB3 is out and it only supports in-process compilations and you can’t wait for the next release? Well, the good news is, FB3 uses the Flex Compiler API. So, all you need is implement a RPC version of the Flex Compiler API; add out-of-process to FB3 as a new SDK; run a server that uses the original flex-compiler-oem.jar and viola…

Since I designed and implemented the API and know the API very well, it’s pretty easy for me to implement the RPC version. In fact, I spent the last few days doing a prototype and IT WORKS! Click the thumbnail image below to see the screenshot.

I hope to polish it up and make it available as soon as possible. If you’re willing to be an early alpha tester, please shoot me an email.

I Don’t See These Tricks In fcsh!

In one of my previous posts, Flex Compiler SHell (fcsh), I mentioned that Flex removed the flex2.tools.Shell class from mxmlc.jar in version 3. For those who’ve read my previous post, flex2.tools.SimpleShell is a simplified version of flex2.tools.Shell and is currently the ‘official’ fcsh. But flex2.tools.Shell is the version I used back then when I worked for Adobe. This does a lot more and I believe you will find that useful. So let me give you a demo here.

I’m going to use the ‘checkinapp’ application in my demo. It is located at flex/sdk/branches/3.0.x/frameworks/tests/checkinapp.

Adobe Flex Compiler SHell (fcsh) Version 3.0.0
Copyright (c) 2006 Adobe Systems Incorporated. All rights reserved.

(fcsh) mxmlc -debug=true -optimize=false checkinapp.mxml
fcsh: Assigned 1 as the compile target id
Loading configuration file frameworks/flex-config.xml
checkinapp/checkinapp.swf (756154 bytes)

To see the dependencies of checkinapp.mxml, we use the ‘dependency’ command. The number ‘1’ is the compile target id. “Top-level Definitions” means the definitions are visible externally to the file. “Inheritance” means they are the superclass and implementing interfaces. “Namespace” means they are the AS3 namespaces found in the application. “Types” means the listed definitions are found in the API signatures. “Expressions” are the definitions that are found elsewhere in the file, e.g. method bodies.

(fcsh) dependency 1 checkinapp.mxml
checkinapp/checkinapp.mxml
===Top-level Definitions===
checkinapp
===Inheritance===
mx.binding:IBindingClient
mx.core:Application
flash.events:IEventDispatcher
===Namespace===
AS3
mx.core:mx_internal
===Types===
mx.events:SliderEvent
ShoppingCart
PanelControl
ThumbnailView
Checkout
mx.styles:CSSStyleDeclaration
mx.utils:ObjectProxy
mx.events:FlexEvent
mx.containers:Panel
flash.events:Event
GridView
mx.effects:Dissolve
mx.containers:Canvas
mx.binding:IWatcherSetupUtil
mx.controls:Image
mx.containers:HBox
ProductDetail
mx.effects:Fade
flash.events:MouseEvent
CartView
mx.controls:HSlider
mx.events:ListEvent
mx.core:UIComponentDescriptor
mx.effects:Resize
mx.binding:Binding
mx.containers:ViewStack
ResizePanelEvent
===Expressions===
mx.controls:LinkButton
_checkinappWatcherSetupUtil
mx.controls:Label
ShoppingCart
mx.controls:Button
mx.events:PropertyChangeEvent
mx.controls:Spacer
PanelControl
ThumbnailView
mx.styles:StyleManager
Checkout
checkinapp_thumbRoll
mx.styles:CSSStyleDeclaration
mx.utils:ObjectProxy
mx.containers:Panel
checkinapp_listRoll
GridView
checkinapp_thumbOff
mx.containers:Canvas
flash.net:navigateToURL
mx.controls:ComboBox
mx.controls:Image
checkinapp_listOn
flash.utils:getDefinitionByName
checkinapp__embed_mxml_assets_products_tg_logo_gif_1172660324
mx.containers:HBox
ProductDetail
mx.containers:VBox
CartView
mx.controls:HSlider
mx.core:UIComponentDescriptor
mx.effects:Resize
mx.containers:ControlBar
checkinapp_listOff
flash.net:URLRequest
mx.binding:Binding
mx.containers:ViewStack
checkinapp_thumbOn

To see the usage of mx.controls:Button, we use the “usage” command. For example, ScrollThumb uses Button as a superclass.

(fcsh) usage 1 mx.controls:Button
[expressions] checkinapp/checkinapp.mxml
[types] framework.swc(mx/controls/scrollClasses/ScrollBar)
[inheritance] framework.swc(mx/controls/scrollClasses/ScrollThumb)
[expressions] checkinapp/ProductDetail.mxml
[expressions] checkinapp/Checkout.mxml
[types] framework.swc(mx/containers/Panel)
[types] checkinapp/CartView.mxml
[types] framework.swc(mx/controls/NumericStepper)
[inheritance] framework.swc(mx/controls/sliderClasses/SliderThumb)
[inheritance] framework.swc(mx/controls/CheckBox)
[types] framework.swc(mx/containers/Accordion)
[types] framework.swc(mx/controls/ComboBase)
[inheritance] framework.swc(mx/controls/RadioButton)
[inheritance] framework.swc(mx/containers/accordionClasses/AccordionHeader)
[types] framework.swc(mx/controls/alertClasses/AlertForm)
[inheritance] framework.swc(mx/controls/LinkButton)
Found: 16

I think many of you have seen and tried the “-link-report” mxmlc command-line option. Personally, I don’t use it because the -link-report XML output could easily reach .5MB in size for even the smallest possible Flex applications. Also, fcsh is better because the “usage-dependency” combo allows you to traverse the dependency graph in either directions.

To see the number of ‘debuggable” lines, we use the “lines” command. The counts below exclude blank lines. Also, for MXML files, the counts do not count XML tag lines, only the lines in <mx:Script>. The most interesting number below, IMO, is the size of the autogenerated code – the percentage is pretty high. The autogenerated codes are mainly for MXML components (e.g. data binding, styles, etc). That tells you one thing: if you were to write those MXML components in AS3, not in MXML, you would have to write a whole lot more. On the other hand, of course, one could argue that the compiler should autogen as little as possible.

(fcsh) lines 1
188 checkinapp/CartView.mxml
697 checkinapp/Checkout.mxml
26 checkinapp/CreditCardPicker.mxml
87 checkinapp/GridView.mxml
129 checkinapp/PanelControl.mxml
191 checkinapp/ProductDetail.mxml
151 checkinapp/ProductThumbnail.mxml
8 checkinapp/ResizePanelEvent.as
27 checkinapp/ShoppingCart.as
325 checkinapp/StatePicker.mxml
106 checkinapp/ThumbnailView.mxml
649 checkinapp/checkinapp.mxml
total number of lines in the source files: 2584
total number of lines in the autogen files: 1424

To see the bytecode size (in bytes) of the definitions, we use the “sizes” command. I’m listing only the top ten classes here. But you know there are hundred of classes in checkinapp.

(fcsh) sizes 1
104300 framework.swc(mx/controls/listClasses/ListBase)
75913 framework.swc(mx/core/UIComponent)
62331 framework.swc(mx/controls/DataGrid)
47073 framework.swc(mx/core/Container)
39207 framework.swc(mx/controls/dataGridClasses/DataGridBase)
39010 framework.swc(mx/controls/listClasses/TileBase)
34284 framework.swc(mx/controls/List)
30588 framework.swc(mx/managers/SystemManager)
28816 checkinapp/checkinapp.mxml
27623 framework.swc(mx/controls/Button)

I think these fcsh commands should help you analyze your codes’ dependencies and sizes and should in turn, minimize your application’s size.

Flex Compiler SHell (fcsh)

In early summer 2005, the Flex development team was busy working on Flex 2. The new Flex compiler redesign and implementation were finally gaining traction (well, you have no idea how frustrating it was when you had to build a new compiler that integrated with some unstable AS3 compiler pieces and had to run those resulting SWFs in an unstable VM). Anyhow, Dan Smith and I believed that it was the right time to discuss with the Flex Builder team about integrating the Flex compiler into FB2.

FB2 is an Eclipse-based IDE for Flex application development. Similar to other IDEs, FB2 should be able to compile Flex applications incrementally. Back then, the new compiler could barely generate good SWFs and the compiler’s full compile performance was not something we were proud of. But you know, things will get better over time, so I decided to switch focus and work a little bit on the incremental compilation support.

Now that the Flex compiler is open source, you should be able to trace how incremental compilation is done. But it might be difficult to identify the code that is responsible for incremental compilation – there is nothing like incrementalCompile() or something like that. One reason is that the compiler algorithms (API.batch1() and API.batch2()) are actually designed for incremental compilations. They treat full compilations as special cases, i.e. a full compilation is a case where everything needs to be recompiled.

Frankly, it wasn’t that hard to nail the basic incremental compilation support as long as you were dealing one unified algorithm for full vs. incremental. But FB at that point did not have any code in it that would invoke the compiler. Well, for testing, I had to write a tool (not just some test cases) to prove that incremental compilation worked.

To simulate how the compiler works in an IDE, what could be better than doing it in some sort of shell environment (Well, writing an Eclipse plugin as a simulator is straightforward too, but that’s not the point!). Yup, a shell. That’s what I built – Flex Compiler SHell (fcsh). Note that it’s SHell, not Shell. Some official Flex documentation are still using ‘Shell’ – I don’t like that. 😛

It took me a couple of days to get fcsh up and running. There are so many upsides using fcsh than mxmlc and/or compc. One of them is that you eliminate JVM startup time and those 2000+ compiler classes that JVM needs to load for even typical Flex applications. Those overhead could add up to 2 seconds. Another advantage is of course, what it’s built for – incremental compilation. I wasted no time dumping mxmlc and compc and used fcsh exclusively for my daily development work.

Soon after its availability, other folks on the team found good use of it. The QA and doc team dramatically cut their time for building their applications by using fcsh to batch-process their apps.

Due to the nature of my work, i.e. compiler development, I needed to be able to get to data that I couldn’t find in a SWF, e.g. dependency info, bytecode sizes, etc. For this reason, I added a slew of fcsh commands:

• abc – outputs abc bytecodes given a definition name
• dependency – displays a list of dependencies given a file name
• files – lists all the source and SWC files that contribute to the resulting SWF
• sizes – lists the bytecode size of all the definitions in the resulting SWF
• lines – displays the number of ‘debuggable’ lines in the resulting SWF
• memory – displays memory usage (when using JDK 1.5)
• mm.cfg – displays settings in mm.cfg
• trace – turns tracing on/off in mm.cfg
• exportorder – displays the export order of the definitions in the resulting SWF

At first, product management and engineering management weren’t warm to the idea of adding a new tool to the SDK without getting requests from our customers. That’s why you don’t see fcsh in the Flex 2 SDK release.

But the Flex product management reversed their decision and shipped fcsh in Flex 2.0.1. That was great news. But the bad news was that the above-mentioned fcsh commands must be removed.

If you still have Flex SDK 2.0.1 installed, you can check out fcsh.jar. I think it points to flex2.tools.SimpleShell. If you change the manifest to point to flex2.tools.Shell, you will get the full version.

If you only have Flex SDK 3 installed, you’re out of luck because flex2.tools.Shell is no longer distributed. In my opinion, the Flex product/engineering management should consider adding the flex2.tools.Shell class back into the distribution.