The Sentence Generator

The Sentences view is a highly configurable tool to generate sentences from a grammar.

When launched from the ABNF editor (using the Tools > Generate Sentences item from the editor's contextual menu), the Sentences view is initialized with the rules of the grammar and all the external grammars referenced:

The view displays three tabs: the Strategies tab, the Start expansions tab, and the Sentences tab. The Strategies tab is used to configure how each rule is expanded to produce parts of the generated sentences. The various strategies are described below. The Start expansions tab displays the sentence patterns from which sentences will be generated. Finally, once strategies have been assigned to the various rules, the Sentences tab is used to display the generated sentences and operate on them.

Configuring the strategies

The sentence generation tool provides a fine-grained control over the way sentences are generated. This is done by assigning each grammar rule a generation strategy. For the purposes of sentence generation, the strategies change the language recognized by a rule. That is why, when a different strategy is chosen for a given rule, the size of the language (the number of sentences) for the rule is automatically updated, as well as the number of sentences for all rules which are directly or indirectly referring to it.

The currently available strategies are:

• All Sentences. All sentences produced by the rule are generated. If the number of sentences is large, this can easily consume all your available memory. Be cautious with this strategy.
• First Sentence (Professional Edition only). Only the first sentence is produced. The ordering of sentences produced by a rule is determined by the left-to-right ordering of the expansions in the rule. For a repeated item, the first sentence is the first sentence generated by the repeated expansion, repeated the minimum number of times. For an alternative, the first sentence corresponds to the first choice.
• Random Sentences (Professional Edition only). A specified number of random direct derivations are chosen among all the direct derivations of the rule. Note that selecting random derivations may lead to the same sentences to be produced multiple times.
  
  When this strategy is selected, three other controls become available:
  • Number of sentences. This is the number of random direct derivations to produce for the rule.
  • Minimum number of sentences per derivation. This number can be set to a value greater than zero to force all direct derivations of the rule to appear in the generated derivations.
  • Minimum probability threshold. When generating a random direct derivation, the sentence generation tool takes into account the number of sentences that can be generated from each derivation. So if a derivation can only lead to a single sentence while another derivation can lead to 9 sentences, the second derivation has ten times more chances of being generated.
    This threshold allows the probability of each derivation to be set to a minimum value. Setting the threshold to 1.0 makes all derivations equally likely, while leaving it to 0.0 lets the algorithm consider only the probability of the derivations. Setting it somewhere between 0.0 and 1.0 is recommanded when one or more derivations can lead to a disproportionate number of sentences.
  
  
• Tags Coverage (Professional Edition only). Produces the smallest set of sentences that covers all the semantic tags in the rule. This strategy can be useful to ensure full coverage of the semantic tags in the rule or grammar.
• All Paths (Professional Edition only). Produces the smallest set of sentences that covers all the paths in the rule. This strategy can be useful to produce sentences covering all the words in a rule or grammar.
• Rule Examples. Uses the @example lines in the rule documentation as the generated sentences. Be careful to keep these sentences in sync with the grammar, as the tool does not validate that these sentence are valid.

Most strategies apply recursively to the rule it is set to and to all its rule references. For example, setting the strategy of the grammar's root rule to Tags Coverage generates sentences which covers all the tags in the root rule and all the rules directly or indirectly referenced. The exceptions are the All Sentences and the Random Sentences strategies. In these cases, the generation algorithm first generate sentences for the rule references using their own strategy, and then generates the resulting sentences by applying the rule strategy locally.

To choose a strategy for a rule, simply select the rule in the tree and choose the appropriate strategy in the drop-down menu on the right:

Multiple rule names can be selected at once. The chosen strategy is then be applied to all the selected rules. Moreover, if a grammar name is selected (like apartment.abnf in the previous image), the strategy is to be applied to all the rules in the grammar. This is very handy for setting a strategy on an imported grammar.

The button in the view's toolbar can be toggled to force (or disable) the execution of semantic tags for each generated sentence. With some semantic tag formats (like swi-semantics/1.0 and the Nuance 9.0 extensions), sentences which first appear to be in grammar (ING) can be made out of grammar (OOG) after executing the semantic tags.

Generating sentences

To start generating sentences based on the selected strategies, simply press the button in the Strategies or Sentences tab:

Since all the sentences are generated before they are displayed, the button in the Sentences tab can be used to stop the generation process if it takes too long to complete.

The generated sentences are displayed in the Sentences tab:

Not all sentences are generated at once. The number of sentences generated at once can be changed using the Sentence Generator preferences.

Note that the number of generated sentences may slightly differ from what one would expect from the configuration of the strategies. This is because if a sentence is generated multiple times, it nonetheless appears only once in the sentence list.

If more sentences can be generated, they can be displayed by pressing the button in the view's toolbar:

The other buttons, when enabled, have the following effect:

• the button adds all the selected sentences to the grammar's coverage testset as in grammar (ING) sentences;
• the button adds all the selected sentences to the grammar's coverage testset as out of grammar (OOG) sentences;
• the button starts the semantics stepper with the selected sentence;
• the button starts the grammar interpreter with the selected sentence;

Sorting

The generated sentences can be sorted lexicographically by pressing the button in the view's toolbar:

Selecting start sentence patterns

Instead of generating sentences starting from the grammar's root rule, a different start rule can be chosen, or even multiple start rules. Simply select the desired rules in the strategies tab and click the Set as start pattern or Add to start patterns hyperlink on the right:

The start patterns are displayed in the Start pattern tab, where they can be removed using the DELETE key:

Alternatively, a more complex sentence pattern can be added from the Sentence Explorer view. The generated sentences all derive from the selected expansion.

Outputting the sentences to a file

The generated sentences can be saved to a file. To activate that feature, select the Add to file automatically check box at the bottom of the view:

Doing this displays a text box where the name of the output file can be typed:

A file selection dialog can also be used to select the file by pressing the ... button on the right.

The sentences are written to the file when they are generated. So it is important to configure that part prior to pressing the Start generation hyperlink or the Next Sentences button.