Near the beginning of last year I moved several of my web applications over to using YUI Builder. Builder provides many advantages over how I was managing the projects before; choosing to use Builder wasnt difficult. Since then, I have hacked, and rewritten parts of Builder so it could meet my project's requirements. Even my modified builder falls short of what I would like it to be. So I'm posting here, after about a year of experience with Builder, a list of improvements that I have already made to my builder and other improvements I would like to see. Since it seems that Buildy is lined up to replace Builder in the near future I'm not going to file enhancement tickets on Builder, unless a YUI developer wants me to, but this is intended to help people improve Buildy or whatever becomes the next generation of Builder.
Please feel free to add your own builder wish list items to this thread or leave any comments you may have.
Wish list items are not listed in any specific order.
Builder is a component build tool. Make a project build tool.
A YUI application of any reasonable complexity will have multiple components. A normal project is set up with a source directory and a build directory. The source directory is what gets checked into a repo and Builder generates the build directory based on the source directory contents. As it is, to build a project, someone must traverse the source directory's subdirectories one by one and call `ant all` in each one, and even then only YUI modules are built. A web application will at least have an index page if not other files to build too. These extra non YUI module build steps will probably be very custom and won't come standard with Builder, but Builder could provide tools and examples to make it easier to create them. Keep the ability to build individual components and add the ability to build an entire project at once. Even more helpful, once an entire project has been built, the next time it gets built, only work on source files that have been modified since the last build.
In my modified builder, my source folder has a build.properties file and a build.ProjectName.properties file. The build.properties file sets up the usual builddir path and output path etc. The build.ProjectName.properties file has configuration settings for the project and how it gets built. By setting certain properties here, I can add/change/remove functionality from different builds of the application. I have also set up different build modes. When building in development mode, the YUI config object is automatically set up not to use a combo service, to use the -debug source files and it will add YUI's test console to the page.
The JsLint step in Builder is pretty much worthless as it is. The JsLint build step should be helpful.
Currently the built raw, -debug, and -min files are run through JsLint and the output is displayed in the console. Since the -min file has to pass, JsLint is set to extremely permissive settings. That alone defeats the purpose for JsLint. Having the output scroll by in the console is not helpful; it's too fast to even notice. Also, since the built files have stuff appended or concatenated to them, it's difficult to trace any errors back to the source code to fix them.
EDIT: Also, lint errors shouldn't fail the build. There are a small handful of lint errors that I choose to ignore. JsLint isn't configurable enough to do exactly what I want it to.
In my modified builder, JsLint is performed on the source files before anything is done to them. JsLint has almost all of settings turned on. (except for variables that start with _) Instead of the output scrolling by in the console, it's written to html files using the same functionality that jslint.com uses. For each source file, one html file gets created. At the end, and index file gets created that loads all the other in iframes kind of like a photo gallery. I can open that index page and quickly scroll through looking for errors; if there is one, I can quickly open that page and the line numbers that it references match the line numbers in my source. There might be an even better way to make JsLint more useful, but this has been working well for me.
Just a side note, I noticed that JsLint modules on NPM seem to be way out of date. JsLint itself seems to get updated and enhanced frequently. Why is there such a lag?
Add a CssLint build step.
Add an HTML Compressor.
HTML compression is extremely tricky to do correctly. I have been using this simple HTML compression algorithm for a while and I haven't noticed any issues:
1: Load the HTML file as a string.
2: Split the string into an array by newlines.
3: Perform ltrim on each line.
4: Join the array without a separator
5: If there is a doctype, insert one newline after it.
.
I usually tab out my HTML with spaces, so this usually gives me a very good compression ratio. It also eliminates all of those extra text nodes in the DOM. It doesn't attempt to remove XML comments or change the markup in any way. If there's some reason you want to have whitespace in your markup, it only does an ltrim so put it on the same line with something else before it.
Remove features: Don't concatenate files within a module. Don't automatically wrap source files with YUI.add.
We have rollups, combo services, virtual rollups, and maybe even remote loader services. There are plenty of ways to concatenate at the module level. I would argue that we do not need to concatenate at the submodule level. I'm not saying that concatenating files within a module is never useful, but I feel like the benefits gained from it are very small compared to the benefits gained by leaving it out.
The first thing we could do once file concatenation is removed is to put the YUI.add wrapper in our source code instead of having builder put it there. Letting Builder wrap the code seems like a convenience, but it is actually a huge hindrance to development. If our source files included YUI.add and the meta data, that would mean that our -debug built files === our source files. To generate the -debug files in the build folder, Builder could simply create symbolic links to the source file. That would make development so much nicer. Currently, any time a source file changes in any way, it has to be built before the results can be tested. That extra step is extremely unfortunate. If the -debug files were symlinks, changes are just a save and a refresh away.
To achieve a similar result with raw and -min files, Builder could be modified to run as a service and monitor the source directory for modifications. It could auto build every time a file is saved.
Also while developing a component, when you realize that you've just introduced a new dependency requirement, it's somewhat annoying to dig through the file system to find the right build.properties file to add the new requirement. It would be much easier to simply scroll down. The amount of stuff defined in build.properties could be reduced.
A neat trick for a build tool written in Node.js: If a js file only has YUI.add calls in it, you can use the vm feature. Define a fake YUI.add method, execute the script with vm, and now you have just read all of the module meta data from the file.
Another note: The YUI.add wrapper that Builder currently puts around module code is wrong. The callback function receives more than one argument.
Add a yuidoc build step.
Add a unit test build step.
Currently it's impossible to individually test private functionality that is wrapped up in closures. Builder could make use of JsDev and generate -test versions of built files.
Builder should generate a YUI_config object.
One of my least favorite things about building a YUI application is maintaining module meta data in two different locations. I frequently forget to keep my master YUI_config object in sync when I'm changing module requirements and then it's not always obvious what went wrong when I test my application and it fails. Builder knows all the requires and file paths for everything, the rest could be defined in build.properties.
There are build time loader optimizations that can be done here as well. For example, if a module requires 'base' and 'widget', Builder could figure out that 'widget' already requires 'base' and remove the redundant requirement. Also, gallery module meta data could be prefetched and baked into YUI_config.
If Builder is rewritten on Node.js, build it with YUI.
Use YUI modules to build YUI modules. It makes sense to me. YUI is an excellent framework for building JavaScript applications, why not use it to make a YUI Builder? This would automatically make it easier for YUI developers to create new build steps or custom build tools. New additions or improvements to builder could be wrapped up as plugins or class extensions and contributed back to the community through the gallery.
Please feel free to add your own builder wish list items to this thread or leave any comments you may have.
Wish list items are not listed in any specific order.
Builder is a component build tool. Make a project build tool.
A YUI application of any reasonable complexity will have multiple components. A normal project is set up with a source directory and a build directory. The source directory is what gets checked into a repo and Builder generates the build directory based on the source directory contents. As it is, to build a project, someone must traverse the source directory's subdirectories one by one and call `ant all` in each one, and even then only YUI modules are built. A web application will at least have an index page if not other files to build too. These extra non YUI module build steps will probably be very custom and won't come standard with Builder, but Builder could provide tools and examples to make it easier to create them. Keep the ability to build individual components and add the ability to build an entire project at once. Even more helpful, once an entire project has been built, the next time it gets built, only work on source files that have been modified since the last build.
In my modified builder, my source folder has a build.properties file and a build.ProjectName.properties file. The build.properties file sets up the usual builddir path and output path etc. The build.ProjectName.properties file has configuration settings for the project and how it gets built. By setting certain properties here, I can add/change/remove functionality from different builds of the application. I have also set up different build modes. When building in development mode, the YUI config object is automatically set up not to use a combo service, to use the -debug source files and it will add YUI's test console to the page.
The JsLint step in Builder is pretty much worthless as it is. The JsLint build step should be helpful.
Currently the built raw, -debug, and -min files are run through JsLint and the output is displayed in the console. Since the -min file has to pass, JsLint is set to extremely permissive settings. That alone defeats the purpose for JsLint. Having the output scroll by in the console is not helpful; it's too fast to even notice. Also, since the built files have stuff appended or concatenated to them, it's difficult to trace any errors back to the source code to fix them.
EDIT: Also, lint errors shouldn't fail the build. There are a small handful of lint errors that I choose to ignore. JsLint isn't configurable enough to do exactly what I want it to.
In my modified builder, JsLint is performed on the source files before anything is done to them. JsLint has almost all of settings turned on. (except for variables that start with _) Instead of the output scrolling by in the console, it's written to html files using the same functionality that jslint.com uses. For each source file, one html file gets created. At the end, and index file gets created that loads all the other in iframes kind of like a photo gallery. I can open that index page and quickly scroll through looking for errors; if there is one, I can quickly open that page and the line numbers that it references match the line numbers in my source. There might be an even better way to make JsLint more useful, but this has been working well for me.
Just a side note, I noticed that JsLint modules on NPM seem to be way out of date. JsLint itself seems to get updated and enhanced frequently. Why is there such a lag?
Add a CssLint build step.
Add an HTML Compressor.
HTML compression is extremely tricky to do correctly. I have been using this simple HTML compression algorithm for a while and I haven't noticed any issues:
1: Load the HTML file as a string.
2: Split the string into an array by newlines.
3: Perform ltrim on each line.
4: Join the array without a separator
5: If there is a doctype, insert one newline after it.
.
I usually tab out my HTML with spaces, so this usually gives me a very good compression ratio. It also eliminates all of those extra text nodes in the DOM. It doesn't attempt to remove XML comments or change the markup in any way. If there's some reason you want to have whitespace in your markup, it only does an ltrim so put it on the same line with something else before it.
Remove features: Don't concatenate files within a module. Don't automatically wrap source files with YUI.add.
We have rollups, combo services, virtual rollups, and maybe even remote loader services. There are plenty of ways to concatenate at the module level. I would argue that we do not need to concatenate at the submodule level. I'm not saying that concatenating files within a module is never useful, but I feel like the benefits gained from it are very small compared to the benefits gained by leaving it out.
The first thing we could do once file concatenation is removed is to put the YUI.add wrapper in our source code instead of having builder put it there. Letting Builder wrap the code seems like a convenience, but it is actually a huge hindrance to development. If our source files included YUI.add and the meta data, that would mean that our -debug built files === our source files. To generate the -debug files in the build folder, Builder could simply create symbolic links to the source file. That would make development so much nicer. Currently, any time a source file changes in any way, it has to be built before the results can be tested. That extra step is extremely unfortunate. If the -debug files were symlinks, changes are just a save and a refresh away.
To achieve a similar result with raw and -min files, Builder could be modified to run as a service and monitor the source directory for modifications. It could auto build every time a file is saved.
Also while developing a component, when you realize that you've just introduced a new dependency requirement, it's somewhat annoying to dig through the file system to find the right build.properties file to add the new requirement. It would be much easier to simply scroll down. The amount of stuff defined in build.properties could be reduced.
A neat trick for a build tool written in Node.js: If a js file only has YUI.add calls in it, you can use the vm feature. Define a fake YUI.add method, execute the script with vm, and now you have just read all of the module meta data from the file.
Another note: The YUI.add wrapper that Builder currently puts around module code is wrong. The callback function receives more than one argument.
Add a yuidoc build step.
Add a unit test build step.
Currently it's impossible to individually test private functionality that is wrapped up in closures. Builder could make use of JsDev and generate -test versions of built files.
Builder should generate a YUI_config object.
One of my least favorite things about building a YUI application is maintaining module meta data in two different locations. I frequently forget to keep my master YUI_config object in sync when I'm changing module requirements and then it's not always obvious what went wrong when I test my application and it fails. Builder knows all the requires and file paths for everything, the rest could be defined in build.properties.
There are build time loader optimizations that can be done here as well. For example, if a module requires 'base' and 'widget', Builder could figure out that 'widget' already requires 'base' and remove the redundant requirement. Also, gallery module meta data could be prefetched and baked into YUI_config.
If Builder is rewritten on Node.js, build it with YUI.
Use YUI modules to build YUI modules. It makes sense to me. YUI is an excellent framework for building JavaScript applications, why not use it to make a YUI Builder? This would automatically make it easier for YUI developers to create new build steps or custom build tools. New additions or improvements to builder could be wrapped up as plugins or class extensions and contributed back to the community through the gallery.