Java Query

May 12th, 2008

This is not linq for Java, but it does bear a small resemblance. Download the source code and unit test. [Update: One unit test requires the Java tuple library.]

If you’ve ever needed to return an Iterable<T> from a Java method, you know that it can be challenging. If you already have exactly the collection that you want to iterate, then all is well. If not, you have two choices.

Option 1, you could construct a new ArrayList<T>, populate it, and return it. This option is less memory efficient than it could be. But it is by far the easiest way to go. You get to write your iteration code using “for”.

Or option 2, you could implement Iterable<T> with an anonymous inner class. Instead of using “for”, you’ll have to manage the state of your iterator yourself. Basically, you have to turn the looping code inside out and put the initialization, testing, and stepping code in separate methods. However, this saves on memory, as you don’t have to store the results in an intermediate collection.

On a number of occasions, I’ve taken option 2. Having rewritten the same ugly code several times, I decided to generalize it.

Inspiration from C#
Having spent some time in the .NET world, I’m familiar with two ways of solving this problem.

First, C# provides the “yield” keyword for turning a loop inside out. You write the code as per option 1, but instead of adding an object to a collection, you “yield return” it. The compiler turns the code inside out and makes it work like option 2. Yael has an implementation of yield return in java, if you like this approach.

Second, C# now has built-in syntax for querying any data source, including collections of objects. This syntax is called linq, or language integrated query. I didn’t want to implement all of the linq functionality in java, but I did use it as a source of inspiration.

From
Linq is like upside-down SQL. The first thing that you write is the “from” clause. This is important for type safety and intellisense. Borrowing from this convention, you start a java query with:

Query
    .from(collection)

This returns a Query<T>, which gives you the rest of the methods.

Where
A where clause filters the collection on-the-fly. The iterator that is produced only lands on the items that satisfy the where condition. You specify the condition by anonymously implementing the Predicate interface, as in:

Query
    .from(collection)
    .where(new Predicate<T>() {
        public boolean where(T row) {
            return /* Condition based on row */;
        }
    })

Join
A join traverses a nested collection. For each object of the first collection, you obtain an iterator over the second. The result is the same as two nested for loops. You return the inner collection using the Relation interface, as in:

Query
    .from(collection)
    .join(new Relation<PARENT, CHILD> () {
        public Iterable<CHILD> join(PARENT parent) {
            return parent.getChildren();
        }
    })

This produces an iterator of Join<PARENT, CHILD>.

Select
The last clause in a query is the select. Now that you’ve built up and narrowed down your collection of objects, the select clause gives you a chance to transform them into the data type that you want. You provide an anonymous implementation of the Selector interface:

Query
    .from(collection)
    .select(new Selector<T, RESULT>() {
        public RESULT select(T row) {
            return row.getProperty();
        }
    })

If you want to get the objects themselves, you can simply use “.selectRow()” instead.

Put it all together
You can combine these clauses to create the query you need. It has to start with a “from” and end with a “select”, but you can have any combination of “join” and “where” clauses in between. Here’s an example from the unit test:

// Iterate through the order items. Pull out order numbers where a widget is ordered.
Iterator<Integer> orderNumbers = Query
    .from(customers)
    .join(new Relation<Customer, Order>() {
        public Iterable<Order> join(Customer parent) {
            return parent.getOrders();
        }
    })
    .join(new Relation<Join<Customer,Order>, Item>() {
        public Iterable<Item> join(Join<Customer, Order> parent) {
            return parent.getSecond().getItems();
        }
    })
    .where(new Predicate<Join<Join<Customer,Order>,Item>>() {
        public boolean where(Join<Join<Customer, Order>, Item> row) {
            return row.getSecond().getPart().getName().equals(“widget”);
        }
    })
    .select(new Selector<Join<Join<Customer,Order>,Item>, Integer>() {
        public Integer select(Join<Join<Customer, Order>, Item> row) {
            return row.getFirst().getSecond().getNumber();
        }
    })
    .iterator();

Now you can generate iterators on-the-fly without wasting memory or writing really ugly code. Under the covers, it’s just doing what a for loop would do. But with a query, you can return the resulting collection to a caller without turning your code inside-out.

Dallas TechFest 2008

May 4th, 2008

The first Dallas TechFest was this weekend. There were sessions on .NET, Ruby, REST, Groovy, Flex, AIR, and many other languages, tools, and platforms. It was a chance for members of different communities to get together and cross-train each other. I made a point of stepping outside my comfort zone, rather than sticking with the .NET track.

But the two sessions that I attended within the .NET track were hosted by Carl Franklin and Richard Campbell of .NET Rocks. I couldn’t miss the chance to see them. The first was a recording of a .NET Rocks episode on community. The second was Richard on web application performance and scalability. If you ever get a chance to hear Richard speak, you must go.

After his talk, I asked Richard to take a look at Update Controls. He was very gracious and gave me at least a half an hour of his time over lunch to show him the demo. He understood the benefits of the library immediately, and gave me some pointers for getting the idea across more effectively. Finally, he told me that I need to complete the database story, which, as it happens, is in the works.

It was a thrill to finally meet Carl and Richard, the guys who inspired me to launch this site. And it was enlightening to see what’s been going on in other parts of the industry. Who knows. Maybe I’ll get a chance to present next year.

The honest EULA

April 15th, 2008

If a EULA actually said what it meant:

This agreement is a legal contract that you (the USER) make to us (the VENDOR) concerning the product (the SOFTWARE).

The SOFTWARE that the USER is about to tolerate is a train wreck in progress. It is difficult to use. It is impossible to configure. It won’t behave as the USER any other rational human being would expect.

In short, it is no different than any other SOFTWARE the USER has ever installed.

The USER will not blame the VENDOR for the behavior of the SOFTWARE, even if the VENDOR would be found criminally negligent or deliberately maniacal. Rather, the USER agrees either to purchase additional SOFTWARE (the UPGRADE) from the VENDOR, or to abandon his data and admit his own foolishness in purchasing the SOFTWARE in the first place.

The USER grants the VENDOR an unlimited license to all information voluntarily entered into the SOFTWARE. If the USER wanted to keep something from the VENDOR, why would he be entering it into the SOFTWARE in the first place? What exactly does the USER have to hide, anyway?

The USER permits the SOFTWARE to contact the VENDOR periodically to transmit data collected about the USER. The USER will not inquire as to the content of the transmission. The USER will not interfere with the transmission. The USER will like it.

The USER agrees to allow the SOFTWARE to install additional products from third parties (the TOOLBAR). The USER will not expect a share of the compensation that the VENDOR collects from the maker of the TOOLBAR. The USER automatically agrees to any EULA attached to the TOOLBAR.

By reading this EULA the USER has already agreed to the terms herein. By ignoring this EULA, the USER is demonstrating his own ignorance and nevertheless agreeing to the terms herein.

Update Controls on Code Project

April 11th, 2008

I wrote a short article for The Code Project called A Simple Alternative to Windows Forms Data Binding. It’s a real quick introduction to Update Controls. The goal here is just to get people trying it out to see how it simplifies Windows UI programming.

It’s only been up for a day, but already I’ve gotten a comment from Marc Clifton himself! If you have spent any time on Code Project, you’ve read at least one of his articles. The man is a brilliant communicator. And he wears a nice hat.

Please check out the article. And if you haven’t done so yet, check out Update Controls .NET.

Relativity and the partial order of facts

April 1st, 2008

Regarding the automotive wireless system that I am currently working on, we are having a discussion about feed reconciliation. It breaks down into two camps: validate everything and reject out-of-order feeds, or accept everything and sort it out later.

In this system, the manufacturer of the wireless device identifies the unit by MEID. They obtain some phone numbers (MDN and MIN) and pair them to the device. They then ship the device to the automaker, who installs it in a vehicle (identified by VIN). Once we have MEID, MDN, MIN, and VIN, we can create an account.

This scenario can be represented with an Entity Fact Diagram. My understanding of the system has changed since this original post, so the diagram is different. A subset of the new diagram follows:

Entity Fact DiagramEntity Fact DiagramWe receive a feed for the pairing of MDN/MIN with MEID. We then receive a feed for the shipment. Finally, we receive a feed from the automaker for the installation of the MEID in a VIN. The installation feed does not include the phone numbers, so we have to have both Pairing and Installation before we create an account.

The argument is what to do when we’ve received an Installation record for an MEID that we haven’t seen in the Pairing feed. It shouldn’t happen, because it takes days for the device to be shipped, received, and installed. Nevertheless, it could happen and probably will.

So the validate everything camp wants to reject Installation records for which Shipment has not been received, and Shipment for which Pairing has not been received. Getting things out of order is indicative of a problem in the overall business process, and should be flagged as early as possible.

Meanwhile, the accept everything camp wants to gather and store all of this input, then run reports to reveal problems. Aging reports will show when things move forward too slowly, and exception reports will show when things skip steps. You still get the feedback, but it stays a business problem and doesn’t become a system problem.

Here’s my solution
The entity fact diagram shows me a hybrid solution. As illustrated, the Pairing feed is a prerequisite of the Shipment feed. The arrow between them has two consequences: first, the Shipment has to have enough information to identify the Pairing. Second, the Shipment cannot exist without the Pairing. This second consequence is of particular importance.

The Pairing and Shipment feeds are from the same source, as indicated by color. These are completely automated feeds produced by a database at the wireless device factory. If we see a Shipment without a Pairing, then it truly is a system problem.

The Installation feed, however, comes from a different source. We have to be more lenient with the relative timing between different sources. If, for example, we are on Einstein’s train moving near the speed of light away from the manufacturer toward the automaker, we will see the Installation lightning strike in front of us before the Pairing strikes behind us. Sure, it seems crazy, but in enterprise software these things happen.

AiS 41: Scrum Doesn’t Suck

March 28th, 2008

Listen Now

Agile is Relative by Scott Ambler.
The Agile Manifesto.
Scrum on Wikipedia.

The WordPress plot thickens

March 27th, 2008

I’ve upgraded WordPress and I’m working on putting things back to normal.

In the process, I’ve discovered a few more side effects of the hack. It appears that the hacker uploaded a new header.php and footer.php to my active theme. I only noticed this because I had changed the theme and placed some important markup in the header. Without this markup, the page doesn’t render correctly. If you are viewing this site on the evening of the 27th, you might just well be seeing the old theme while I work on fixing the new one.

The footer was modified to include hidden links to various link farm pages on a health and fitness podcast. The header was modified to include only one link, this to a blog run by someone by the name of Erik Kastner. I won’t give you the links because … well … that’s the point.

I’ve posted a comments on these blogs asking for help in figuring out this connection. I’ll let you know the response.

Update
Please see Erik’s reply in the comments. My current working theory is that the hidden links in the footer — obviously placed to improve SEO, not to gather direct links — are the main payload of this attack. I suspect that the hidden link (Erik’s link) in the header points to the next site that the attacker plans to compromise. This forwards the page rank and makes the payload on the next page more valuable. Erik’s blog runs on WordPress as well, but his was patched.

So thanks, Erik. Let’s hope that I get a quick response from the health and fitness podcast.

Forensic analysis of WordPress version change

March 27th, 2008

It happened again. The db_version was reset to 1 this morning, causing the error message “Your database is out-of-date”.

My forensic analysis turned up nothing. Whatever caused this also had the effect of disabling the Spam Karma plugin, so all of the comments posted thereafter were sitting in moderation. This gave me a good timestamp for the occurrence.

Since the version change disables the spam filter, I suspected an attack. Perhaps someone takes advantage of a WordPress bug to turn off the filter and then post a bunch of comment spam. So I checked the server log for any suspicious entries around the time of the earliest comment in moderation.

The earliest comment was at 22:56 GMT. My server is located in the -400 time zone, so I scanned the access log for 18:56. I can see a “POST /blog/wp-comments-post.php” at that time, so that is the first comment post with the spam filter turned off. Scrolling back to the previous post at 18:43, I have my window. The version change and disabling of Spam Karma occurred somewhere in there.

At 18:45, there are several POSTs to various suspicious URLs.

  • /blog/wp-admin/options.php
  • /blog/wp-admin/upload.php
  • /blog/wp-admin/inline-uploading.php?post=-1&action=upload
  • /blog/wp-admin/upload.php?style=inline&tab=upload&post_id=-1

Unfortunately, I cannot see the body of the HTTP POST from the access log, so I don’t know exactly what was done. But it appears that someone used these entry points to change some options and upload a file called “ro8kfbsmag.txt”. Googling this filename, I found that it is indeed an attack.

It looks like this vulnerability has been fixed. So the solution is to upgrade to the latest version of WordPress. I know what I’ll be doing tonight!

Cheap Zyrtec
Tramadol Hcl
Zyloprim
Phentermine No Prescription
Elavil
Information credit report
Lamisil
Highest credit score
Protonix
Shell credit card application
Amoxil
Eminem Ringtones
Phentermine Hcl
Online Cialis Jelly
Ipsec VPN
Buy Xenical
Information credit report
Nexium
Alprazolam Online
Remind Ringtones
Repair credit score
Alltel Ringtones
Credit reports com
Chase redit card srevices
3 credit report
Baclofen
Cheap Casodex
Viramune
Cheap Zithromax
Cheap Neurontin
Buy Elavil
Fosamax
Lotensin
Credit score ranges
Sumycin
Buy Aristocort
Adipex
Free Ringtones
Disputing credit reports
Counseling credit card debt
Ativan Online
Commercial credit report
Debt consolidation credit cards
Credit report
Online Herbal Phentermine
Deltasone
Viagra Alternatives
Nokia Ringtones
Buy Synthroid
Buy Augmentin
Free Mp Ringtones
Check credit report
Credit score system
Cheap Lasix
Buy Levitra
Altace
Cheap Viagra Jelly
Credit score in
Cheap Cephalexin
Credit card offer
Phentrimine
Cleaning up credit report
Free credit report scores
Buy Ultram
Cialis Dosage
Credit report monitoring service
Credit card deal maker
Buy Prevacid
Ativan Drug
Tylenol
Buy Obestat
Cialis Soft Tabs
Nexium
Cheap Protonix
Cheap Viagra Soft Tabs
Linksys VPN
Online Zithromax

Your database is out-of-date

March 24th, 2008

For some reason, WordPress started throwing the error “Your database is out-of-date. Please upgrade.” whenever I visited the admin console. I had not upgraded either the code or the database prior to getting this error. I have no clue what caused it.

So naturally, I Googled the error message and found many other people with the same problem. Some of these people received replies to their cries for help. The replies included such helpful tidbits as “so upgrade, already”, to “reboot the server”. Naturally, I tried them and none worked for me.

Since WordPress is written in PHP, the source code is entirely in the clear. This makes it possible for a typical software blogger like myself to debug the application and figure out what’s wrong. I used PuTTY to log into my server and peruse the code and database to see what could be done. I thought this would make for gripping television, so I did it all while streaming.

Here’s my solution
I found that a single value in the wp_options table had been changed. It should be 4772, but it was 1.

If you grep for the error message, you will find a comparison between get_option(’db_version’) and $wp_db_version. Further grepping will reveal that $wp_db_version is set in wp-includes/version.php and that get_option selects option_value from the wp_options table. Log into mysql and select * from wp_options where option_name=’db_version’ and you will find the culprit.

This solution is not for the weak of heart. If you are not familiar with mysql, please take some time to get comfortable with it before messing around with your own blog. You run the risk of seriously messing things up, especially if you forget your where clause!

If you are not familiar with PHP, however, not to worry. This fix does not require any changes to the WordPress PHP code. In fact, this exercise is a great way to learn.

So if you found this post by Googling your error message, watch the video clip, get your PuTTY ready, and fix it!

BTW, how did the value get changed? Not a clue. If it happens again, I’ll investigate further. And I’ll stream the process!

Update Controls .NET released under LGPL

March 18th, 2008

I have released Update Controls .NET as an open source project.

Update Controls is a set of .NET Windows Forms controls that update themselves automatically. Instead of setting properties, you handle events. For example, the GetText event fires on an UpdateTextBox when it needs to update itself. The event returns the text string to display. Then, when any data that was used to calculate that string changes, the event fires again. Automatically.

Think of how you use a spreadsheet. You enter a formula in a cell, with references to other cells. When those other cells change, the formula is recalculated. You don’t have to tell it to update, it just figures out when to do so.

I created the algorithm at the core of Update Controls almost ten years ago in C++. Since then, I’ve translated it first into Delphi, then Java, and finally C#. The Update Controls library is that core algorithm applied to the common Windows Forms controls. It also includes a handful of themed controls for better visual appeal.

Not data binding
For as long as Microsoft has done demos of VB, they’ve tried to convince us that we can write real software with zero lines of code. It works fine on the projector screen, but it fails to translate to a full-scale application. Once you get outside the box of direct column/control relationships, it is impossible to adapt data binding to your needs.

But Update Controls is completely flexible. You can start with a simple event that returns one value, but then modify it to combine two or more values as your needs change. Since you are writing real code, you are in complete control.

A data-bound control attaches directly to a column in the database, or a property of an object in the data model. It cannot call through more complex business logic.

But an Update Control can call any business logic that you specify. Whatever data the business logic touches in order to do its work, that data affects the control’s behavior. When any of it changes, the event fires and calls the business logic again.

Not the observer pattern
The observer pattern is obsolete. It requires that you manually register each observer with each subject. To make it work, you have to know ahead of time all of the objects in the data model that affect each user interface component. A large portion of your source code is dedicated to making and breaking those connections.

Update Controls, on the other hand, discovers those dependencies on its own. It can see through business logic, so you don’t have to directly tie your user interface to your data model. And if that business logic switches to a new path, the dependencies are updated as well.

The observer pattern also suffers from cascading redundant updates. If your observer depends upon two subjects and both are changed at the same time, the observer pattern will give you two updates. The problem gets even worse when you have indirect dependencies, as the redundancy rolls through the system like so many dominoes.

But Update Controls queues the updates until the last possible moment, so they occur only once. Even indirect dependencies are calculated in the best possible order, with each update being invoked just once as the information bubbles to the surface.

Please give it a try
The library works in C# and VB. It can be used in either Visual Studio 2005 or 2008, with .NET 2.0, 3.0, or 3.5. You can download the source code and build it yourself, or just run the convenient installer. Videos, demos, and documentation are all available to help you get started.

Once you write a Winforms app with Update Controls, you’ll never want to code without them.