Extracting Tasks from Emails: first challenges

Borderline Cases

Ambiguity

Task requests tend to be ambiguous in the sense of placing a direct or indirect obligation to a recipient. Take the following sentences for example:

  • I need your report by the end of the Week
  • I would like to get the report by the end of the Week
  • We need to finish the report by the end of the Week
  • Hope you can send me the report by the end of the Week
  • Hope you didn’t spend too much additional time on this

Conditionality

Some cases involve conditional clauses with the format “if this, then that”. Sometimes they are task related while other cases are not.

  • If you could send me your report soon, that would be appreciated
  • If you need more information, please call John

Requests for Inaction

Another common case is sentences that are asking for inaction or require the user to avoid doing something. The recipient of the email should not feel any obligation to perform something. However, it is common to find statements that are using a negated form and indeed are being used to request something:

  • Please do not reply to this e-mail
  • Please do not forget to send me your report

“Let me know…” pleasantries

Perhaps one of the most common border cases are polite requests that make use of the pattern “let me know…”:

  • Please let me know if you have any questions”.
  • Please let me know if you are going to attend
  • Please let me know the name of the contact to send the report

Third-Party Requests

Another interesting case is when a request places an obligation to a third party, considering that a textual conversation within an email can involve more than one recipient:

  • John, please call our accountant to set up a meeting
  • My assistant, John Doe, will call you to set up a meeting

Attachment Review Requests

Finally, there are also many common statements in emails that simply inform of the presence of an attachment, but in a manner that can be interpreted as a request: “Please find attached a copy of the report.” In such cases it is hard to determine the degree of obligation without knowing more context:

  • Please see the attached invitation
  • Attached please find my resume for your review

Labelling Data

Taking into consideration the borderline cases described above, we defined our own annotation guidelines to help us decide if a candidate sentence was a task or not. Remember that this manually annotated data is intended to serve as input to train a ML classifier. Making sure that annotations are as clean and correct as possible is crucial in ML, because any trained model based on wrong labelled data would replicate the same mistakes as the human annotators. Defining what is a task and what is not a task, and having a clear agreement about it before labelling data, allowed us to make annotation decisions with a higher degree of reliability.

Inter-annotator agreement over a corpus of 2000 sentences

Conclusions

We can conclude that our inter-annotator results show a reasonable level of agreement if we take into account the complexity of the problem, including all the linguistic, stylistic, pragmatics and polite variations that can be used to express requests in conversational emails.

References

[1] Lampert, A., Paris, C. and Dale, R. (2007). Can Requests-for-Action and Commitments-to-Act be Reliably Identified in Email Messages? In Proceedings of the 12th Australasian Document Computing Symposium, pp 48–55, Melbourne, Australia.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store