# Basics of text extraction ## In this section * parsing Akoma Ntoso XML using Python's LXML library * extracting all text in a naive way * ignoring editorial remarks ## XML vs HTML Extracting text from Akoma Ntoso XML is the preferred process. You can also extract text from the HTML version of an AKN document, but it's a little different because HTML is not as explicitly structured as XML. We'll be using the [Cape Town Liquor Trading by-law](https://openbylaws.org.za/akn/za-cpt/act/by-law/2014/control-undertakings-liquor/eng@2014-12-12) for these examples. ## Parsing Akoma Ntoso XML Let's fetch the raw XML from the Laws.Africa API. ```python from urllib.request import Request, urlopen TOKEN = "your-auth-token" url = "https://api.laws.africa/v3/akn/za-cpt/act/by-law/2014/control-undertakings-liquor/eng/.xml" request = Request(url, headers={"Authorization": f"Token {TOKEN}"}) # raw_xml is a bytes object, not a str raw_xml = urlopen(request).read() print(raw_xml[:100]) # b'

Example of Akoma Ntoso XML, including metadata and text.

The simplest way of extracting text is to ignore the structure and just extract all text nodes.

# itertext() iterates over all text nodes in the entire document
text = ' '.join(root.itertext())
print(text[:100])
# To provide for the control of undertakings selling liquor to the public including the control of tra

This is quick and easy, but includes **all** text nodes, which may not be what we want: * It includes text from all structural elements including headings, numbers, editorial comments, footnotes, quoted and embedded elements, tables, etc. * It includes all portions of the document, including attachments such as schedules and appendixes. ## Ignoring editorial remarks When legislation is edited or amended, sometimes editorial remarks are added. For example:

<p eId="sec_6__subsec_4__p_1">
  <remark status="editorial">
    [subsection (4) deleted by section 1 of the <ref href="/akn/za-cpt/act/by-law/2014/control-undertakings-liquor-amendment" eId="sec_6__subsec_4__p_1__ref_1">Amendment By-law, 2014</ref>]
  </remark>
</p>

These are not substantive and not officially part of the legal text of the document. We usually want to exclude these remarks when extracting text for full-text search purposes. To do this, we can use [XPath](https://en.wikipedia.org/wiki/XPath), a powerful mechanism of querying elements in XML documents. {% hint style="info" %} Learn more about XPath at these resources: * [Zyte's XPath tutorial for web scraping](https://www.zyte.com/blog/an-introduction-to-xpath-with-examples/) * [Using lxml and XPath to extract text in Python](https://lxml.de/tutorial.html#using-xpath-to-find-text) {% endhint %} XPath is rich and expressive and we will only cover the basics for this example. There are two key ideas we need: * A **namespace** lets us ignore parts of the document that aren't part of the Akoma Ntoso XML standard. The [official Akoma Ntoso XML namespace](https://docs.oasis-open.org/legaldocml/akn-core/v1.0/akn-core-v1.0-part2-specs.html) is `http://docs.oasis-open.org/legaldocml/ns/akn/3.0` * The **query** which describes the elements we want from the XML document. In this case, our query will mean "text nodes that aren't part of `` elements". ```python # The AKN namespace is the default one for this XML document, # which is given by the None entry in the namespace map (nsmap). # This is the same as ns = "http://docs.oasis-open.org/legaldocml/ns/akn/3.0" ns = root.nsmap[None] # This tells xpath that when we use the "a" alias, we mean the AKN namespace. # It saves us from writing the full namespace in the xpath query. nsmap = {"a": ns} # query all the text nodes that don't have as an ancestor text = ' '.join(root.xpath("//text()[not(ancestor::a:remark)]", namespaces=nsmap)) print(text[:100]) # To provide for the control of undertakings selling liquor to the public including the control of tra ``` Here's a brief explanation of what the components of the XPath query mean:

XPath	Meaning
`//`	Any node at any point in the tree. Alternatively, `./` means nodes at the current element (which is `root` in this case), or `/` which means the root of the entire document.
`text()`	This matches text nodes.
`[...]`	This applies additional conditions to the text nodes being matched. The conditions must evaluate to true to be included in the resulting node set.
`not(...)`	This negates the condition inside the brackets.
`ancestor::a:remark`	The `ancestor::` means any ancestor element of the text node, and `a:remark` limits the ancestors to those that are `<remark>` elements in the Akoma Ntoso namespace.

## Ignoring other content Other types of text content you may want to ignore, depending on your use case: * Text of headings and sub-headings: ``, ``, `` * The numbers of chapters, sections, etc.: `` * Quoted and embedded content: ``, `` Depending on your needs, you can adjust the XPath to include these additional elements by including them in the `not(...)` clause and separating them with `or`. ```python # get text that isn't in a remark, heading or num text = ' '.join(root.xpath( "//text()[not(ancestor::a:remark or ancestor::a:num or ancestor::a:heading)]", namespaces=nsmap)) ``` In the next section, we'll explore how to extract text only for certain portions of the document, such as a particular section or chapter, or only table elements.