Guest Post: HFCD 4 Review

Hello, my name is Jeremy Petersen.  I’m the Vice President of Technology at School Improvement Network, and I wanted to take a moment to write a review of my experiences with HFCD 4.

Let me start with a little background. The School Improvement Network  http://www.schoolimprovement.com provides the solutions, products, and services that help teachers and educators increase student achievement.  Our flag ship product is a large Adobe Flex application called PD 360.  PD 360 is an on-demand professional learning system for teachers and educators, with a series of tools built around a library of video segments.

PD 360 started out as a simple on demand video player application back with Flex 2.0 in the fall of 2006.  The application has followed the maturing of the Flex platform with a full Flex 3 rewrite, and is now in the process of a full Flex  4 rewrite.   Along with the obvious advances and growth inherited in keeping up with latest versions of Flex/Flash Builder, the application has also grown exponentially in complexity, size, and features.  In fact, our Flex 4 rewrite has a (insert expletive) 42 modules!  The project has also gone from a couple of developers that tinkered on it in their spare time to team comprising of 4 full time Flex engineers, and 2 full time ColdFusion engineers.

Let me start out by saying that we love Flex 4, but after making the switch from Flex 3 and having rearchitected our application from the ground up (enter the 42 modules), we definitely noticed a HUGE hit in compile times.  All of our engineers have very capable hardware (Corei7 etc).  But to even change the text on a single button in the application would result in a 4-5 minute compile time.  Talk about a drop in productivity.

Now, enter HFCD 4.  One of our engineers read about it on a blog and decided to try it out.  His results were too good to be true.  After waiting a couple of weeks to make sure he did not have any issues, the rest of the team changed over on 30 day trial licenses to test things out.  Remember that 4-5 minute compile time from before?  With HFCD 4 , making changes to a single module, that compile time is down to 4-5 seconds!  Granted changes to code that is linked in to multiple modules (say our controller etc) still take a couple of minutes to compile, but for most of the work our team is doing we are officially back in business of writing code and not reading blogs or firing up World of Warcraft in between every single run/ compile.

So it goes without saying that we love this product!  But my review does not stop here. I have one other thing to comment on.  The support of Clement Wong (HFCD creator).  Clement went above and beyond our expectations in support and answering questions.  For some reason my personal install had some issues (it ended up being Eclipse problems).  Clement not only helped me via email, but even offered to do a phone call to get me up and running.  In addition, he is very responsive to feature requests and suggestions.

HFCD comes with a free 30 day trial, so grab a copy and try it out.  You have nothing to lose and everything to gain!

Impressive Hardware Setup For Running HFCD

Recently I’ve received an email from a Flex developer. He told me that his development team has been using HFCD for Flex development. What impressed me was his team hardware setup. Basically, his team of developers connect Flex Builder running on their laptops to multiple instances of HFCD running on a powerful LAN server. He is quite happy about the improved development experience.

The server we’re sharing for compilation is a Dell box with basic hardware config:

OS: Solaris 10
CPU: Dual quad-core Xeon 2.8ghz (two threads per core — OS reports 16 cores)
Memory: 32GB
Ethernet: 4 connected Ethernet connections (mutliplexed between Solaris Zones)

This box hosts a few zones that do various other things too (host db’s, app servers), however all for dev purposes so it’s never generally that busy.

As you might guess, using HFCD on this box to compile has significant advantages to the laptops we develop on.

Awesome! 🙂

HFCD zip Arhives for Ant Lovers

Some developers don’t use Flex Builder and wonder how they can use HFCD from the command line or with their favorite Flex IDEs.

I have just made some HFCD zip archives available:

http://bytecode-workshop.com/hfcd/3/download/hfcd_3.2.0_full.zip
http://bytecode-workshop.com/hfcd/3/download/hfcd_3.3.0_full.zip
http://bytecode-workshop.com/hfcd/3/download/hfcd_3.4.1_full.zip
http://bytecode-workshop.com/hfcd/3/download/hfcd_3.5.0_full.zip
http://bytecode-workshop.com/hfcd/4/download/hfcd_4.0.0_full.zip
http://bytecode-workshop.com/hfcd/4/download/hfcd_4.1.0_full.zip

You don’t have to download all of them… just pick the version that you intend to use.

Unzip the archive. You’ll find a build.xml inside. Edit ${flex.sdk.dir} to point to your corresponding Flex SDK installation directory. Run ant. After running build.xml, you should see the HFCD “client” directory and HFCD “server” directory.

You start HFCD by running server/bin/hfcd. If you want to use ant to call HFCD, use client/lib/hfcd-ant-tasks.jar. For more information on how to use the HFCD ant tasks, please check out the HFCD ant task language reference:

http://bytecode-workshop.com/hfcd/doc/HFCD-Ant-Task-Language-Reference.pdf

Some developers want to see Maven/Flexmojos support. I like that idea too and I’m going to investigate. Stay tuned.

HFCD For Flex 4.1 Now Available

Dear readers,

HFCD for Flex 4.1 is now available. Sorry for the delay. I thought it would take me a week or two to release it. But I wasted a little bit of time working on the wrong Flex SDK revision! Boo…

The good news is that there is no Flex Compiler API changes in Flex 4.1. However, there will be some new methods added to the API in the upcoming Flex 4.5.

I think the most notable fix in the compiler in 4.1 is SDK-25206. If you occasionally switch between airglobal.swc and playerglobal.swc in your Flex Builder workspace, you may run into this issue. But this is a good fix and helps build performance too.

http://bytecode-workshop.com/#section=downloads

HFCD 4ドキュメントは現在日本語で入手可能です。

このたびHFCD 4の日本語版ドキュメントが、ご利用いただけるようになりましたのでお知らせします。

• HFCD 4 インストール・ガイド
• HFCD 4 ユーザー・ガイド
• HFCD Ant タスク言語のリファレンス

HFCD For Flex 4.1 Coming Soon…

I started working on an HFCD update for Flex 4.1 yesterday. Looks like it will not be a lot of work for me so I plan to make it available early next week.

If you would like to know the progress, you can follow me on (@stopcoding) Twitter.

HFCD 4 Is Now Officially Available.

Hi, I’m pleased to announce that HFCD 4 is now officially available.

HFCD 4 for Flash Builder achieves better Flex (full and incremental) build performance than Flex 3 and Flex 4 by utilizing:

1. all of the Flex 4 compiler performance improvements
2. multicore processor technology
3. techniques to build Flex apps in the background.

and allows builds to be distributed to multiple machines on a local network. The HFCD ant tasks also make it easy to integrate HFCD with custom build systems or popular build systems like Hudson CI, etc.

To learn more about HFCD, please visit http://bytecode-workshop.com/.

Those who purchased HFCD 3 before today will receive their FREE copies of HFCD 4. Sorry, this special offer ends today.

But don’t be disappointed. If you find a HFCD bug and file it and I can reproduce it, you will get a HFCD license for free. Click here for details. The offer ends by the end of this month (2010/04) but I’m going to extend it until further notice…

Download HFCD: http://bytecode-workshop.com

Using Apache Ant To Run Parallel HFCD Builds

In several of my previous posts about the use of the HFCD ant tasks,

• HFCD Ant Tasks Outperform Adobe Flex Ant Tasks…
• Ant Tasks for HFCD Now Available
• Coming Soon: Ant Tasks for HFCD

I only mentioned in the build.xml example that you can connect to one HFCD server instance. What I did not mention was that you could script your build.xml to connect to multiple HFCD server instances by using the Apache Ant <parallel> task.

    <parallel>
        <hfcd hostname="host1" buildData="${basedir}/build.hfcd.1">
            ....
        </hfcd>
        <hfcd hostname="host2" buildData="${basedir}/build.hfcd.2">
            ....
        </hfcd>
        <!-- more hfcd tasks... -->
    </parallel>

The build.xml code snippet above shows the use of the Apache Ant <parallel> task with the <hfcd> task. If you are able to separate your extremely large project workspace into multiple pieces (e.g. project workspace with a large number of modules), you could use the above technique to distribute the build to multiple HFCD server instances, further improving overall build performance.

I did not mention it then because the above technique was not possible without being able to store the build data sent from multiple HFCD server instances separately. But now, it is possible. In the latest HFCD build (i.e. build 20100420), I added the buildData tag attribute to the <hfcd> task. The tag attribute specifies the directory where build data from a HFCD server instance should be stored.

The latest HFCD build is here:

http://bytecode-workshop.com/#section=downloads

The HFCD Ant Task Language Reference is here:

http://bytecode-workshop.com/hfcd/doc/HFCD-Ant-Task-Language-Reference.pdf

More On The Incredible Shrinking JVM Heap…

In the previous post, I talked about the unusual phenomenon of seeing JVM shrink its heap and returning unused memory back to the operating system when running HFCD. But I didn’t talk much about how it happened. This post is to follow up on that…

First of all, the two key JVM parameters responsible for resizing JVM heap are:

• -XX:MinHeapFreeRatio (default is 40)
• -XX:MaxHeapFreeRatio (default is 70)

According to Sun (now Oracle),

By default, the virtual machine grows or shrinks the heap at each collection to try to keep the proportion of free space to live objects at each collection within a specific range. This target range is set as a percentage by the parameters -XX:MinHeapFreeRatio=<minimum> and -XX:MaxHeapFreeRatio=<maximum>, and the total size is bounded below by -Xms and above by -Xmx .

It goes on to say:

With these parameters if the percent of free space in a generation falls below 40%, the size of the generation will be expanded so as to have 40% of the space free, assuming the size of the generation has not already reached its limit. Similarly, if the percent of free space exceeds 70%, the size of the generation will be shrunk so as to have only 70% of the space free as long as shrinking the generation does not decrease it below the minimum size of the generation.

Both Flash Builder and HFCD use the default -XX:MinHeapFreeRatio and -XX:MaxHeapFreeRatio setting. As a matter of fact, by default, HFCD only sets -Xmx1024m and lets JVM tune itself.

Now take a look at the heap usage of Flash Builder when running the compiler internally. It looks obvious that the requirement for -XX:MaxHeapFreeRatio is not met. Therefore, no shrinking.

But take a look a the heap usage of HFCD. The used heap (blue line) drops so much (i.e. free space well exceeds 70% at the end) that JVM is able to resize the heap downward based on the default -XX:MaxHeapFreeRatio setting.

I also tried -server (HotSpot Server VM) and -client (HotSpot Client VM) on HFCD and manage to get the downward heap resizing to kick in.

However, there are two caveats you need to know:

1. By default, JVM uses serial GC. If you start JVM by using the parallel GC (i.e. -XX:+UseParallelGC), you will NOT see this kind of huge downward resizing. I could be wrong but I think it’s because the parallel GC uses the adaptive size policy to control the size of the generations and ignore the -XX:MaxHeapFreeRatio setting.
2. The charts above were generated by running Java 5/6 on Windows. The JVM on Mac OSX does adjust heap size downward from time to time but they are all very minor downward adjustments and it does not do that major downward resizing after finishing a full build.

There you go.

It Is Nice To See JVM Shrink Its Heap…

Every Flex developer and their mother knows that the Flex compiler uses a lot of memory to build applications! That’s not good… Fortunately, Adobe continues to work diligently to reduce the compiler’s memory usage. That being said, we still have to deal with what we currently have. We all know that once you get JVM to grow the heap to build your applications, it is tricky to make it give the memory back to the operating system. Take a look at the following example, I use Flash Builder to build the Flex framework projects that are setup in the Flex open source branch.

Flash Builder, at peak usage, uses just above 460 MB of memory to run a full build. After a forced garbage collection, the memory reading stays there… pretty much unchanged.

Of course, using the Windows Task Manager is hardly scientific. So I switch to Java Visual VM.

The orange line shows that the JVM heap does not shrink at all after a forced garbage collection. That kinda corroborates with the reading in the Windows Task Manager. There are many reasons why JVM does not shrink the heap. One obvious reason is that the used heap (blue line) stays relatively high there.

Now, I reconfigure Flash Builder to use HFCD to build the same Flex framework projects. The Windows Task Manager shows that the HFCD memory usage, at its peak, uses about 395MB of memory. But the memory usage of HFCD plunges to 80MB after running a full build. HFCD gives a huge chunk of memory back to the operating system!!

The Java Visual VM again, corroborates with the reading in the Windows Task Manager. Note that the orange line falls off a cliff after a full build. The JVM heap shrinks to a level well under 100MB!

If you are in the “memory-is-cheap” camp, you probably will not care. But it’s a big deal if you want to run some other applications during the day and when Flash Builder is holding onto a large chunk of system memory.

I will talk a bit more about this topic in the coming week.

BTW, I’ve updated HFCD 4 this morning. It includes several bug fixes and some minor performance enhancements. The good news is that HFCD 4 is still a lot faster than the FB built-in compiler in most medium-to-large FB workspaces.

You can download HFCD here (http://bytecode-workshop.com/) Remember, if you buy a copy of HFCD 3 now, you will receive a copy of HFCD 4 for FREE when it becomes officially available.