Validate for People

Lately, I seem to have found more web forms that are doing a ridiculously crappy job of validating the input data. It has, on occasion, gotten me yelling at my computer because I can’t understand what the owners of the site were thinking. Some of them are bad enough that they make me stop using the site, assuming that’s an option.

The other day, I had to enter a phone number. My password manager entered the phone number as

630.555.1212

I submitted the form. The app told me my phone number was invalid. I, manually, change the phone number to:

630-555-1212

I submitted the form again. Again, the app told me my phone number was invalid. I read the entire error message this time. I switched my phone number to:

6305551212

I submitted the form and, voila, the stupid application finally accepted it.

On my banks website, I would paste in payment amounts. Most of the time, those amounts were formatted (e.g. $1,234.55). When I submitted the payment, my bank would tell me I entered an invalid dollar amount. I had to remove the dollar sign and the comma to get them to accept the amount. (My bank has since fixed this problem).

Also on my banks website, I have some bills go straight to the bank. Occasionally, I’ll submit a payment for a bill the bank received, making no changes. When I submit the payment, the site tells me I provided an invalid date. That makes me yell since I didn’t provide the date. I then realize the date falls on a non-banking day so I fix it and submit the payment again.

These problems have one big thing in common. The web app can’t recognize valid data and it is making this my problem. It shouldn’t be my problem the site  can’t recognize a valid phone number. It shouldn’t be my problem the site can’t recognize a valid dollar amount. It shouldn’t be my problem that the site can’t fix a date or identify the difference between an actual invalid date and a data that is not an option.

Validations are necessary to ensure an app can do stuff with the data. Sometimes that stuff is for me; sometimes it is for the owner of the app. Validations are critical to keeping applications running.

But validations that require following a strict formatting guide are not. Make validations flexible. Computers are good at parsing data. Support all valid formats. Support unformatted input. Fix data before you display it. Make things easy for your user. Because a happy user is the best reference you can get.

 

The Need for Technical Discipline

Technical discipline is a critical part of any development team. To me, technical discipline is the act/process of continually improving the code. In other words, pay down technical debt as it is created. Doing this can mean the difference between an enjoyable project and a hated project.

Every story completed by every developer adds technical debt. Sometimes the debt is realized immediately; other times the debt appears because of a  future change request. The problem with debt is it makes change more difficult. And, the more debt that is allowed to build up, the more difficult changes become.

When the debt builds to some critical mass is when the project is in trouble. Simple changes that used to take 2 days take twice as long. Complex changes are taking three or four times as long. It’s difficult to explain why.

This is when the pressure to deliver stories starts to weigh down on everyone. And everybody gets unhappy because this great project has just become the latest debacle.

How Did We Get Here?

Most projects don’t start intending to build up lots of technical debt. Usually the road to having mountains of technical debt is paved with good intentions.

 

It starts when with small things. Sometimes these sacrifices start with a decision to deliver a time-sensitive item quickly, sacrificing whatever we have too. More commonly, it is a lot of little decisions made by the developers as they are developing stories:

• Avoiding refactoring work to hit the estimate
• Limiting automated testing to hit the estimate
• Not removing unused code
• Leaving Ugly, complicated code as-is
• Duplicating common code instead of extracting it

For a more thorough explanation of the types of technical debt, check out http://martinfowler.com/bliki/TechnicalDebtQuadrant.html.

When the pile of technical debt gets big enough, the developers try to start doing it. But this leads to a different problem. The business team starts to pressure the developers to cut out those unnecessary activities because delivery is getting slower. After all, if it was okay to avoid these things up until now, why isn’t it okay now? And, if the development team gives into that pressure, things get worse on the project quickly.

The development team has to realize that they are mostly responsible for this situation. But avoiding things that extended the time early on, precedent has been set that the activities are totally unnecessary.

One way I’ve seen attempts to fix this is to create refactoring stories. That is, create a story to fix the mess made while developing a story. I’ve seen lots of projects with refactoring stories but I’ve never seen a project fix a technical debt problem with refactoring stories. The approach doesn’t work because the refactoring stories, aka stories that take valuable development time and deliver nothing, have a way of always having the lowest priorities.

How Do We Fix It? And Keep It From Recurring?

The fix is simple but hard to implement. Don’t cut corners during development except when there are pressing reasons to do so. And those exceptions should be few. In other words, the developers should strive to slightly improve the code base with every story.

This is a difficult transition. The longer the team has let discipline lapse, the more difficult it is. After just a few painful weeks, the team should start seeing improvements in delivery times. Things will continue to improve; slowly, for quite a while.

When making the transition, it’s important to not try and fix everything at once, that’s impossible. Instead, focus on the areas of the codebase that cause the most difficulty with the stories being played.  Other areas will be addressed later when they are being changed by other stories.

When technical discipline lapses, the development team can feel like it is being a better business partner by delivering faster. The reality is the opposite, delivering faster means  paying dearly later.Being the better partner means retaining discipline all the time.

Change – Embrace It

I’m a developer. I’ve been a developer for more years than I care to think about. Thinking back, I realize that how I code today is different enough from what I did after graduating,  that it is almost a different profession.

I started writing COBOL on a mainframe. That COBOL had SQL embedded in it to talk to the database. CICS commands were also added to send and get data from the user.

Today, I write code in a few different languages. My current project uses c#, Javascript, HTML, CSS and a large number of libraries like React, Entity Framework, and Unity (Unity for IOC).

When I look at the work at a low level, not much has changed:

• I’m still using a relational database to save and retrieve data
• Data continues to be presented by “screens”
• Those “screens” are stateless

But as soon as you above that lowest level, nothing is the same:

• Database activity is done through an ORM; there’s very little SQL coding
• Getting data to and from the server is taken care of by libraries; this only shows up as some attributes/markers within the code
• Development environments provide instant feedback about compilation errors
• Automated tests identify what broke almost instantly

When you look at the applications themselves, there is no resemblance between the applications I built 20 years ago and the ones I build today. To me, the changes are profound.

Getting from then to today was challenging. The change I struggled with the most was the move to TDD. It was such a different take on how I was used to approaching development that I struggled quite a bit. But once I got it, I realized what it’s power and it has influenced every aspect of my development, even when I’m not following TDD. I also think it was the single biggest personal improvement I have made. And I regret that I didn’t start it sooner.

And that’s how some changes go. You adopt them and slowly realize that it’s a significant improvement in how you do thing. But not all changes are like this. Some, don’t amount to much. Others, are not a fit at all. The most difficult category though, are those changes that just fail because they were never a good idea.

• 4GL (4th Generation Languages) were going to eliminate the need for developers; they would allow business users to code their own programs. While some of the 4GL concepts have made it into the mainstream, this whole idea has disappeared from the market.
• The Client-Server model was the direction of the future. It was a way to move off of the mainframe and provide better interfaces to the business users. Then the scaling issues became apparent and all but killed this approach. While it continues to exist as layered architectures deployed on servers, the original approach has all but disappeared.

Change is difficult. Some changes are more difficult than others. But we need to embrace change because changing is what allows us to continually improve our skills. And that means we can deliver better software. And that is what we all want.

Exception Handling Anti-Patterns

Exception handling seems easy. But, done poorly, it can cause problems of its own or worse, it can cause problems that hide other problems.

Eating Exceptions

The worst anti-pattern is eating exceptions. This is when an exception is caught, nothing (or minimal logging) is done to handle the exception, and processing continues.

try {
... Do Stuff ...
}
catch {
Debug.WriteLine("Oh no, an exception occurred.")
}

This pattern causes some of the hardest to find bugs. Especially when the try block calls code in other classes.

Why are these bugs so nasty? The failure occurs leaving an object (or objects) in an invalid state. The failure is ignored and processing continues assuming everything is good. This can lead to corrupted data, and other, hard to reproduce exceptions.

Finding these exceptions is onerous; usually requiring some luck as well as skill.

What’s the fix for this pattern? 99+% of the time , the best fix is to remove the try/catch block. Let the exception be handled by the application-level handlers; let the stop once an object is in an invalid state.

One of the first things I do when I start on a new codebase is to remove try/catch blocks that are not:
1. Application-level exception handlers
2. Handling an exception
This sometimes causes some angst from the other developers. But it will also slowly (or, sometimes, not so slowly) expose those hard-to-find bugs that have been lurking so that they can be fixed.

Losing Exception Information

Another common problem is losing track of some of the critical information about the exception, most commonly, the stack trace.

Let’s assume we are doing some database update code, and we need to handle exceptions to issue a rollback.

var transaction = new Transaction(connection);
try {
    ... update the database ...
    transaction.Commit();
}
catch (Exception e) {
    transaction.Rollback();
    throw new Exception($"Issued rollback {e.Message}");
}

In this example, the stack trace is lost. Does that matter?

It can when you are trying to find and fix the bug. Assume you are using Entity Framework and SQL Server. One common exception is an EntityException (or a descendant) with the message:

String or binary data would be truncated. The statement has been terminated.

Finding the problem behind this error is a challenge in the best of circumstances. Finding it without knowing which update caused the problem is that much more difficult.

Fortunately, fixing this problem is easy. Instead of appending the original exception message, send the entire exception

var transaction = new Transaction(connection);
try {
    ... update the database ...
    transaction.Commit();
}
catch (Exception e) {
    transaction.Rollback();
    throw new Exception($"Issued rollback", e);
}

Now, assuming the application level exception handler provides stack traces, finding the statement that caused the problem is easy.

And, like the previous anti-pattern, I search for references to the Message property of any exception when working on a new codebase. And I refactor to pass the entire exception because the information being lost is too important.