User Generated Forms with WTForms

As part of my past work with the Office for National Statistics (ONS), I worked with the survey runner team to add additional features to their Electronic Questionnaire which enabled the business to take regular surveys electronically via the web. During my final months there, I took it upon myself to tackle what was seen to be one of the projects major pieces of technical debt, its use of a custom form renderer. This was identified early on as a feature that shared a lot in common with existing form libraries, but our current renderer wasn’t able to be easily isolated from the rest of the system and was therefore pushed onto a debt backlog after a possible approach had been prototyped.

Due to the hard work of the entire team, we’d made some strategic changes to the system that meant this was possible in my later months. WTForms was the library we wanted to use, but we needed a number of features which weren’t documented or supported out of the box. Chief among these was the ability to create forms dynamically, informed by a JSON description.

Dynamic form definition

WTForms generally expects metaclasses that describe form definitions, as per the following example from their docs:

class MyForm(Form):
  first_name = StringField(u'First Name', validators=[validators.input_required()])
  last_name  = StringField(u'Last Name', validators=[validators.optional()])

I found it easier to think as these definitions as ‘schemas’ for form classes. Typically you’d use them to describe a class which doesn’t change, so WTForms use of them for describing our dynamic forms led to some problems.

When the instantiated form is passed to jinja and rendered as part of a template helper call, appropriate inputs will be rendered for each field definition, with names corresponding to the attribute names (first_name, last_name). We can then take the post request form and pass as an argument to our form class to validate during a submission.

def register(request):
  form = MyForm(request.POST)
  if request.method == 'POST' and form.validate():
      user = MyForm()
      user.first_name = form.first_name.data
      user.last_name = form.last_name.data
      user.save()
      redirect('register')
  return render_response('register.html', form=form)

The approach taken for the ONS was to read in a json description from file and set field attributes dynamically within a method upon what was essentially an empty form class. I’ve simplified the actual project code below to remove detail outside the example.

def get_answer_fields(question, data):
    answer_fields = {}
    for answer in question['answers']:
        name = answer.get('label') or question.get('title')
        answer_fields[answer['id']] = get_field(answer, name)
    return answer_fields

def generate_form(json_for_page, data):

    class QuestionnaireForm(Form):
        answer_fields = {}

        for question in SchemaHelper.get_questions(json_for_page):

            answer_fields.update(get_answer_fields(question, data))

            for answer_id, field in answer_fields.items():
                setattr(QuestionnaireForm, answer_id, field)

        if data:
            form = QuestionnaireForm(MultiDict(data))
        else:
            form = QuestionnaireForm()

        return form

Here you see that we retrieve details of the json structure for a survey page, and maintain a map of answer fields to be set on the class. The answer ids within the page structure will always be unique and is used as the key for the mapping to each field type. In this way, the resultant form is a collection of fields able to be driven dynamically from our json description.

Validating Dynamic Forms

As well as the fields on our forms being customisable, the validators used upon them were too. This meant for instance, that some fields were optional or required and the messages used for validation needed to be informed by the survey’s json description.

Below you can see an example of the message for a DateRequired validator being updated based on the message from the schema. You can also see an example of modifying the behaviour of the validator, based on whether or not the date validator includes a day attribute as part of the data sent. This allows for the same validator to be used for different form representations of dates.

class DateRequired(object):
     def __init__(self, message=None):
         if not message:
             message = error_messages['MANDATORY']
         self.message = message

     def __call__(self, form, field):
         if hasattr(form, 'day'):
             if not form.day.data and not form.month.data and not form.year.data:
                 raise validators.StopValidation(self.message)
         else:
             if not form.month.data and not form.year.data:
                 raise validators.StopValidation(self.message)

In more complex situations, it was necessary to pass custom data to our validators to later check against. You can see below, when validating custom date ranges, we pass the ‘to’ date and initialise the validator with it, before the call to the validator is made with the form (in this case a subform for just a date).

class DateRangeCheck(object):
    def __init__(self, to_field_data=None, messages=None):
        self.to_field_data = to_field_data
        if not messages:
            messages = error_messages
        self.messages = messages

    def __call__(self, form, from_field):

        if form.day and form.month and form.year and self.to_field_data:
            to_date_str = "{:02d}/{:02d}/{}".format(int(self.to_field_data['day'] or 0), int(self.to_field_data['month'] or 0),
                                                    self.to_field_data['year'] or '')
            from_date_str = "{:02d}/{:02d}/{}".format(int(form.day.data or 0), int(form.month.data or 0),
                                                      form.year.data or '')

            from_date = datetime.strptime(from_date_str, "%d/%m/%Y")
            to_date = datetime.strptime(to_date_str, "%d/%m/%Y")

            date_diff = to_date - from_date

            if date_diff.total_seconds() == 0:
                raise validators.ValidationError(self.messages['INVALID_DATE_RANGE_TO_FROM_SAME'])
            elif date_diff.total_seconds() < 0:
                raise validators.ValidationError(self.messages['INVALID_DATE_RANGE_TO_BEFORE_FROM'])

We actually found it necessary to go one step further in that we needed to halt validation for forms in surveys which were considered optional. The solution was an OptionalForm, allowing empty forms to be considered valid if they had no content.

class OptionalForm(object):
     """
     Allows completely empty form and stops the validation chain from continuing.
     Will not stop the validation chain if any one of the fields is populated.
     """
     field_flags = ('optional',)

     def __call__(self, form, field):
         empty_form = True

         for formfield in form:
             has_raw_data = hasattr(formfield, 'raw_data')

             is_empty = has_raw_data and len(formfield.raw_data) == 0
             is_blank = has_raw_data and len(formfield.raw_data) >= 1 \
                 and isinstance(formfield.raw_data[0], string_types) and not formfield.raw_data[0]

             # By default we'll receive empty arrays for values not posted, so need to allow empty lists
             empty_field = True if is_empty else is_blank

             empty_form &= empty_field

         if empty_form:
             raise validators.StopValidation()

Final Thoughts

One of the main pain points with WTForms adoption was the lack of form-level validation supported out of the box. We had for instance a number of custom forms, which fell outside the generic examples above, where it was necessary to override the validate method on the form itself. This added complexity to the form that I felt belonged in a validator, similar to the field level ones WTForms supports defined in separate classes. Adding to option of form validators would be great and cover most of the bases and seems like an oft-requested feature.

I felt that the use of WTForms really helped create a far more structured approach to form design, creation and validation within our project. Getting to the point where we could isolate and shift it over to a common library was a real achievement.