Partition - Aryn Documentation

Please find the documentation for the Aryn SDK Partition module below. All parameters are optional unless specified otherwise.

Synchronous Partitioning Functions

partition_file

Sends a file to Aryn DocParse and returns a Python dictionary with elements containing its document structure and text.

Parameters

file: Required. A file opened in binary mode to parse or a path expressed as a str or a PathLike object specifying where the file to parse is.
threshold: A float between 0.0 and 1.0, inclusive, which serves as a cutoff to determine which bounding boxes for the file are returned or a string auto (the default) where the service uses a processing method to find the best prediction for each possible bounding box. Only bounding boxes that the model predicts with a confidence score higher than the threshold specified will be returned. A lower value will include more objects but may have overlaps, while a higher value will reduce the number of overlaps but may miss legitimate objects. If you do set the threshold manually, we recommend starting with a value of 0.32.
extract_table_structure: A boolean that, when True, enables DocParse to extract tables and their structural content using a purpose built table extraction model. If set to False, tables are still identified but not analyzed for their structure; as a result, table cells and their bounding boxes are not included in the response. Default is False.
text_mode: A string that specifies the mode to use for text extraction. The default is standard, which extracts the embedded text. The other options are fine_grained, which extracts embedded text in a more fine_grained manner, standard_ocr which uses the classical OCR pipeline, and vision_ocr which uses a vision model for OCR. Note that vision_ocr is only available for PAYG users only.
text_extraction_options: A map with string keys specifying options for text extraction.
- ocr_text_mode (deprecated): A string that specifies the mode to use for OCR text extraction. The default is standard, which uses the conventional classical OCR pipeline to process documents. The other option is vision, which uses a vision model for OCR. Note that vision is only available for PAYG users only.
- remove_line_breaks: A boolean that specifies whether to remove line breaks from the text. Default is False.
table_extraction_options: A map with string keys specifying options for table extraction. Only applied when extract_table_structure is True. Default is empty ({})
- include_additional_text: Boolean. When True, DocParse will attempt to enhance the table structure by merging in tokens from text extraction. This can be useful for working with tables that have missing or misaligned text. Default is False
- model_selection: String. An expression to instruct DocParse how to select the table model to use for extraction. Default is "pixels > 500 -> deformable_detr; table_transformer", which means “if the largest dimension of the table is more than 500 pixels, use deformable_detr; otherwise use table_transformer.” To use only deformable_detr or table_transformer, set model_selection="deformable_detr" or model_selection="table_transformer". Selection expressions are of the form
  metric cmp threshold -> model; metric cmp threshold -> model; model
  And should be read as a series of if metric compares to threshold, then use model statements. Statements are processed from left to right.
  - Supported models are table_transformer, which tends to do well with smaller tables, and deformable_detr, which tends to do better with larger tables.
  - Supported metrics are pixels, which corresponds to the maximum dimension of the bounding box containing the table (we find this to be easier to reason about than the total number of pixels which depends on two numbers), and chars, which corresponds to the total number of characters within the table as determined by the OCR/text extraction step.
  - Thresholds must be numeric.
  - Supported comparison operators are <, >, <=, >=, ==, !=.
  A statement with no metric, comparison, and threshold can be thought of as a default, where statements after the default will not be processed. If no such ‘unconditional’ statement is included and no conditions match, DocParse will default to table_transformer. Anything after the unconditional statement will not be processed. Examples:
  - table_transformer => always use table transformer
  - pixels > 500 -> deformable_detr; table_transformer => if the biggest dimension of the table is greater than 500 pixels use deformable detr. Otherwise use table_transformer.
  - pixels>50->table_transformer; chars<30->deformable_detr;chars>35->table_transformer;pixels>2->deformable_detr;table_transformer;comment => if the biggest dimension is more than 50 pixels use table transformer. Else if the total number of chars in the table is less than 30 use deformable_detr. Else if there are mode than 35 chars use table transformer. Else if there are more than 2 pixels in the biggest dimension use deformable detr. Otherwise use table transformer. comment is not processed.
extract_images: A boolean that determines whether to extract images from the document. The format is determined by the value of extract_image_format. Default: False.
extract_image_format: A string indicating what in what format extracted images should be returned. Must be one of ppm, png, or jpeg. In all cases, the result will be base64 encoded before being returned. Default: ppm.
summarize_images: (PAYG Only) A boolean that, when True, generates a summary of the images in the document and returns it as the text_representation. When False, images are not summarized. Default is False.
selected_pages: A list specifying individual pages (1-indexed) and page ranges from the document to partition. Single pages are specified as integers and ranges are specified as lists with two integer entries in ascending order. A valid example value for selected_pages is [1, 10, [15, 20]] which would include pages 1, 10, 15, 16, 17 …, 20. selected_pages is None by default, which results in all pages of the document being parsed.
chunking_options: A dictionary of options for specifying chunking behavior. Chunking is only performed when this option is present, and default options are chosen when chunking_options is specified as {}.
- strategy: A string specifying the strategy to use to combine and split chunks. Valid values are context_rich and maximize_within_limit. The default and recommended chunker is context_rich as {'strategy': 'context_rich'}.
  - Behavior of context_rich chunker: The goal of this strategy is to add context to evenly-sized chunks. This is most useful for retrieval based GenAI applications. Context_rich chunking combines adjacent section-header and title elements into a new section-header element. Merges elements into a chunk with its most recent section-header. If the chunk would contain too many tokens, then it starts a new chunk copying the section-header to the start of this new chunk and continues. Merges elements on different pages, unless merge_across_pages is set to False.
  - Behavior of maximize_within_limit chunker: The goal of the maximize_within_limit chunker is to make the chunks as large as possible. Merges elements into the last most recently merged set of elements unless doing so would make its token count exceed max_tokens. In that case, it would keep the new element separate and start merging subsequent elements into that one, following the same rule. Merges elements on different pages, unless merge_across_pages is set to False.
- max_tokens: An integer specifying the cutoff for splitting chunks that are too large. Default value is 512.
- tokenizer: A string specifying the tokenizer to use when determining how characters in a chunk are grouped. Valid values are openai_tokenizer, character_tokenizer, and huggingface_tokenizer. Defaults to openai_tokenizer.
- tokenizer_options: A nested dictionary with string keys specifying the options for the chosen tokenizer. Defaults to {'model_name': 'text-embedding-3-small'}, which works with the OpenAI tokenizer.
  - Available options for openai_tokenizer:
    - model_name: Accepts all models supported by OpenAI’s tiktoken tokenizer. Default is “text-embedding-3-small”
  - Available options for HuggingFaceTokenizer:
    - model_name: Accepts all huggingface tokenizers from the huggingface/tokenizers repo.
  - character_tokenizer does not take any options.
- merge_across_pages: A boolean that when True the selected chunker will attempt to merge chunks across page boundaries. Does not apply to the mixed_multi_column merger, which never merges across pages. Defaults to True.
output_format: A string controlling the output representation. Defaults to json which yields an array called elements which contains the partitioned elements, represented in JSON. If set to markdown the service response will instead include a field called markdown that contains a string representing the entire document in Markdown format.
output_label_options: A dictionary of options to specify which heuristic to apply to enforce certain label outputs. If this option is not specified, no heuristic is applied. The options the dictionary supports are listed below.
- promote_title: A boolean that specifies whether to promote an element to title if there’s no title in the output.
- title_candidate_elements: A list of strings that are candidate elements to be promoted to title.
- orientation_correction: A boolean value that specifies whether to correct the orientation of rotated pages during the preprocessing step.
markdown_options: A dictionary of options to specify what to include in the markdown output.
- include_pagenum: A boolean that specifies whether to include page numbers in the markdown output. Default is False.
- include_headers: A boolean that specifies whether to include headers in the markdown output. Default is False.
- include_footers: A boolean that specifies whether to include footers in the markdown output. Default is False.
ssl_verify: A boolean that controls whether the client verifies the SSL certificate of the chosen DocParse server. ssl_verify is True by default, enforcing SSL verification.
aryn_config: An ArynConfig object (defined in aryn_sdk/config.py), used for finding an api key. If aryn_api_key is set it will override this. The default ArynConfig looks in the env var ARYN_API_KEY and then in the file ~/.aryn/config.yaml. Default is None (aryn-sdk will look in the aryn_api_key parameter, in your environment variables, and then in ~/.aryn/config.yaml).
aryn_api_key: An Aryn API key, provided as a string. You can get one for free at aryn.ai/get-started. Default is None (If not provided, the sdk will check for it in the environment variable ARYN_API_KEY or will look in aryn_config as specified above).
use_ocr (deprecated): A boolean value that, when set to True, causes DocParse to extract text using an OCR model. This is useful when the text is not directly extractable from the PDF, such as when the text is part of an image or when the text is rotated. When set to False, DocParse extracts embedded text from the input document. Default is False.

Example

from aryn_sdk.partition import partition_file

with open("my-favorite-pdf.pdf", "rb") as f:
    data = partition_file(
        f,
        aryn_api_key="MY-ARYN-API-KEY",
        text_mode="standard_ocr",
        extract_table_structure=True,
        extract_images=True
    )
elements = data['elements']

Return Value

Exceptions

Aynchronous Partitioning Functions

partition_file_async_submit

Submit a document for asynchronous partitioning and get its task_id. The results of the task will remain available in the system for 48 hours. Meant to be used with partition_file_async_result. Note: sending multiple asynchronous partitioning tasks at the same time does not guarantee that they will run simultaneously.

Parameters

file: Required. A file opened in binary mode to parse or a path expressed as a str or a PathLike object specifying where the file to parse is.
threshold: A float between 0.0 and 1.0, inclusive, which serves as a cutoff to determine which bounding boxes for the file are returned or a string auto (the default) where the service uses a processing method to find the best prediction for each possible bounding box. Only bounding boxes that the model predicts with a confidence score higher than the threshold specified will be returned. A lower value will include more objects but may have overlaps, while a higher value will reduce the number of overlaps but may miss legitimate objects. If you do set the threshold manually, we recommend starting with a value of 0.32.
extract_table_structure: A boolean that, when True, enables DocParse to extract tables and their structural content using a purpose built table extraction model. If set to False, tables are still identified but not analyzed for their structure; as a result, table cells and their bounding boxes are not included in the response. Default is False.
text_mode: A string that specifies the mode to use for text extraction. The default is standard, which extracts the embedded text. The other options are fine_grained, which extracts embedded text in a more fine_grained manner, standard_ocr which uses the classical OCR pipeline, and vision_ocr which uses a vision model for OCR. Note that vision_ocr is only available for PAYG users only.
text_extraction_options: A map with string keys specifying options for text extraction.
- ocr_text_mode (deprecated): A string that specifies the mode to use for OCR text extraction. The default is standard, which uses the conventional classical OCR pipeline to process documents. The other option is vision, which uses a vision model for OCR. Note that vision is only available for PAYG users only.
- remove_line_breaks: A boolean that specifies whether to remove line breaks from the text. Default is False.
table_extraction_options: A map with string keys specifying options for table extraction. Only applied when extract_table_structure is True. Default is empty ({})
- include_additional_text: Boolean. When True, DocParse will attempt to enhance the table structure by merging in tokens from text extraction. This can be useful for working with tables that have missing or misaligned text. Default is False
- model_selection: String. An expression to instruct DocParse how to select the table model to use for extraction. Default is "pixels > 500 -> deformable_detr; table_transformer", which means “if the largest dimension of the table is more than 500 pixels, use deformable_detr; otherwise use table_transformer.” To use only deformable_detr or table_transformer, set model_selection="deformable_detr" or model_selection="table_transformer". Selection expressions are of the form
  metric cmp threshold -> model; metric cmp threshold -> model; model
  And should be read as a series of if metric compares to threshold, then use model statements. Statements are processed from left to right.
  - Supported models are table_transformer, which tends to do well with smaller tables, and deformable_detr, which tends to do better with larger tables.
  - Supported metrics are pixels, which corresponds to the maximum dimension of the bounding box containing the table (we find this to be easier to reason about than the total number of pixels which depends on two numbers), and chars, which corresponds to the total number of characters within the table as determined by the OCR/text extraction step.
  - Thresholds must be numeric.
  - Supported comparison operators are <, >, <=, >=, ==, !=.
  A statement with no metric, comparison, and threshold can be thought of as a default, where statements after the default will not be processed. If no such ‘unconditional’ statement is included and no conditions match, DocParse will default to table_transformer. Anything after the unconditional statement will not be processed. Examples:
  - table_transformer => always use table transformer
  - pixels > 500 -> deformable_detr; table_transformer => if the biggest dimension of the table is greater than 500 pixels use deformable detr. Otherwise use table_transformer.
  - pixels>50->table_transformer; chars<30->deformable_detr;chars>35->table_transformer;pixels>2->deformable_detr;table_transformer;comment => if the biggest dimension is more than 50 pixels use table transformer. Else if the total number of chars in the table is less than 30 use deformable_detr. Else if there are mode than 35 chars use table transformer. Else if there are more than 2 pixels in the biggest dimension use deformable detr. Otherwise use table transformer. comment is not processed.
extract_images: A boolean that determines whether to extract images from the document. The format is determined by the value of extract_image_format. Default: False.
extract_image_format: A string indicating what in what format extracted images should be returned. Must be one of ppm, png, or jpeg. In all cases, the result will be base64 encoded before being returned. Default: ppm.
summarize_images: (PAYG Only) A boolean that, when True, generates a summary of the images in the document and returns it as the text_representation. When False, images are not summarized. Default is False.
selected_pages: A list specifying individual pages (1-indexed) and page ranges from the document to partition. Single pages are specified as integers and ranges are specified as lists with two integer entries in ascending order. A valid example value for selected_pages is [1, 10, [15, 20]] which would include pages 1, 10, 15, 16, 17 …, 20. selected_pages is None by default, which results in all pages of the document being parsed.
chunking_options: A dictionary of options for specifying chunking behavior. Chunking is only performed when this option is present, and default options are chosen when chunking_options is specified as {}.
- strategy: A string specifying the strategy to use to combine and split chunks. Valid values are context_rich and maximize_within_limit. The default and recommended chunker is context_rich as {'strategy': 'context_rich'}.
  - Behavior of context_rich chunker: The goal of this strategy is to add context to evenly-sized chunks. This is most useful for retrieval based GenAI applications. Context_rich chunking combines adjacent section-header and title elements into a new section-header element. Merges elements into a chunk with its most recent section-header. If the chunk would contain too many tokens, then it starts a new chunk copying the section-header to the start of this new chunk and continues. Merges elements on different pages, unless merge_across_pages is set to False.
  - Behavior of maximize_within_limit chunker: The goal of the maximize_within_limit chunker is to make the chunks as large as possible. Merges elements into the last most recently merged set of elements unless doing so would make its token count exceed max_tokens. In that case, it would keep the new element separate and start merging subsequent elements into that one, following the same rule. Merges elements on different pages, unless merge_across_pages is set to False.
- max_tokens: An integer specifying the cutoff for splitting chunks that are too large. Default value is 512.
- tokenizer: A string specifying the tokenizer to use when determining how characters in a chunk are grouped. Valid values are openai_tokenizer, character_tokenizer, and huggingface_tokenizer. Defaults to openai_tokenizer.
- tokenizer_options: A nested dictionary with string keys specifying the options for the chosen tokenizer. Defaults to {'model_name': 'text-embedding-3-small'}, which works with the OpenAI tokenizer.
  - Available options for openai_tokenizer:
    - model_name: Accepts all models supported by OpenAI’s tiktoken tokenizer. Default is “text-embedding-3-small”
  - Available options for HuggingFaceTokenizer:
    - model_name: Accepts all huggingface tokenizers from the huggingface/tokenizers repo.
  - character_tokenizer does not take any options.
- merge_across_pages: A boolean that when True the selected chunker will attempt to merge chunks across page boundaries. Does not apply to the mixed_multi_column merger, which never merges across pages. Defaults to True.
output_format: A string controlling the output representation. Defaults to json which yields an array called elements which contains the partitioned elements, represented in JSON. If set to markdown the service response will instead include a field called markdown that contains a string representing the entire document in Markdown format.
output_label_options: A dictionary of options to specify which heuristic to apply to enforce certain label outputs. If this option is not specified, no heuristic is applied. The options the dictionary supports are listed below.
- promote_title: A boolean that specifies whether to promote an element to title if there’s no title in the output.
- title_candidate_elements: A list of strings that are candidate elements to be promoted to title.
- orientation_correction: A boolean value that specifies whether to correct the orientation of rotated pages during the preprocessing step.
markdown_options: A dictionary of options to specify what to include in the markdown output.
- include_pagenum: A boolean that specifies whether to include page numbers in the markdown output. Default is False.
- include_headers: A boolean that specifies whether to include headers in the markdown output. Default is False.
- include_footers: A boolean that specifies whether to include footers in the markdown output. Default is False.
ssl_verify: A boolean that controls whether the client verifies the SSL certificate of the chosen DocParse server. ssl_verify is True by default, enforcing SSL verification.
aryn_config: An ArynConfig object (defined in aryn_sdk/config.py), used for finding an api key. If aryn_api_key is set it will override this. The default ArynConfig looks in the env var ARYN_API_KEY and then in the file ~/.aryn/config.yaml. Default is None (aryn-sdk will look in the aryn_api_key parameter, in your environment variables, and then in ~/.aryn/config.yaml).
aryn_api_key: An Aryn API key, provided as a string. You can get one for free at aryn.ai/get-started. Default is None (If not provided, the sdk will check for it in the environment variable ARYN_API_KEY or will look in aryn_config as specified above).
use_ocr (deprecated): A boolean value that, when set to True, causes DocParse to extract text using an OCR model. This is useful when the text is not directly extractable from the PDF, such as when the text is part of an image or when the text is rotated. When set to False, DocParse extracts embedded text from the input document. Default is False.

webhook_url: a string url for Aryn to visit when the async task has stopped. POSTs a body like this: {"done": [{"task_id": "aryn:t-47gpd3604e5tz79z1jro5fc"}]}

Example

from aryn_sdk.partition import partition_file_async_submit

with open("my-favorite-pdf.pdf", "rb") as f:
    response = partition_file_async_submit(
        f,
        text_mode="standard_ocr",
        extract_table_structure=True,
    )

# get task id
task_id = response["task_id"]

Return Value

Exceptions

partition_file_async_result

Gets the results of an asynchronous partitioning task by task_id. Meant to be used with partition_file_async_submit.

Parameters

Example

import time
from aryn_sdk.partition import partition_file_async_result

# Poll for the results
while True:
    result = partition_file_async_result(task_id)
    if result["task_status"] != "pending":
        break
    time.sleep(1)

Return Value

A dict like the one in the example below containing “task_status”. When “task_status” is “done”, the returned dict also contains “result” which contains what would have been returned had partition_file been called directly. If there is an error with partitioning the file itself, then the “task_status” will still be “done” but the “result” will contain an “error” field indicating what went wrong.

“task_status” can be “done” or “pending”.

{
  "task_status":"done",
  "result": ...
}

Exceptions

partition_file_async_cancel

Cancels the task associated with the task_id specified.

Parameters

Example

from aryn_sdk.partition import partition_file_async_submit, partition_file_async_cancel
task_id = partition_file_async_submit(
            "path/to/file.pdf",
            text_mode="standard_ocr",
            extract_table_structure=True,
            extract_images=True,
        )["task_id"]

partition_file_async_cancel(task_id)

Return Value

Exceptions

partition_file_async_list

Lists all the partition_file tasks still running in your account.

Parameters

Example

Return Value

A dict like the one below which maps task_ids to a dict containing details of the respective task.

{
    "aryn:t-sc0v0lglkauo774pioflp4l": {
        "task_status": "pending"
    },
    "aryn:t-b9xp7ny0eejvqvbazjhg8rn": {
        "task_status": "pending"
    }
}

Exceptions

Helper Functions

convert_image_element

Convert an image element to a more usable format. If no format is specified, create a PIL Image object. If a format is specified, output the bytes of the image in that format. If b64encode is set to True, base64-encode the bytes and return them as a string.

Parameters

Example

from aryn_sdk.partition import partition_file, convert_image_element

with open("my-favorite-pdf.pdf", "rb") as f:
    data = partition_file(
        f,
        extract_images=True
    )
image_elts = [e for e in data['elements'] if e['type'] == 'Image']

pil_img = convert_image_element(image_elts[0])
jpg_bytes = convert_image_element(image_elts[1], format='JPEG')
png_str = convert_image_element(image_elts[2], format="PNG", b64encode=True)

Return Value

Exceptions

draw_with_boxes

Create a list of images from the provided PDF, one for each page, with bounding boxes detected by the partitioner drawn on.

Parameters

Example

from aryn_sdk.partition import partition_file, draw_with_boxes

with open("my-favorite-pdf.pdf", "rb") as f:
    data = partition_file(
        f,
        aryn_api_key="MY-ARYN-API-KEY",
        text_mode="standard_ocr",
        extract_table_structure=True,
        extract_images=True
    )
pages = draw_with_boxes("my-favorite-pdf.pdf", data, draw_table_cells=True)

Return Value

Exceptions

table_elem_to_dataframe

Create a pandas DataFrame representing the tabular data inside the provided table element. If the element is not of type table or doesn’t contain any table data, return None instead.

Parameters

Example

from aryn_sdk.partition import partition_file, table_elem_to_dataframe

with open("partition-me.pdf", "rb") as f:
    data = partition_file(
        f,
        text_mode="standard_ocr",
        extract_table_structure=True,
        extract_images=True
    )

# Find the first table and convert it to a dataframe
df = None
for element in data['elements']:
    if element['type'] == 'table':
        df = table_elem_to_dataframe(element)
        break

Return Value

tables_to_pandas

For every table element in the provided partitioning response, create a pandas DataFrame representing the tabular data. Return a list containing all the elements, with tables paired with their corresponding DataFrames.

Parameters

Example

from aryn_sdk.partition import partition_file, tables_to_pandas

with open("my-favorite-pdf.pdf", "rb") as f:
    data = partition_file(
        f,
        aryn_api_key="MY-ARYN-API-KEY",
        text_mode="standard_ocr",
        extract_table_structure=True,
        extract_images=True
    )
elts_and_dataframes = tables_to_pandas(data)

Return Value

Aryn SDK Reference

​Synchronous Partitioning Functions

​partition_file

​Aynchronous Partitioning Functions

​partition_file_async_submit

​partition_file_async_result

​partition_file_async_cancel

​partition_file_async_list

​Helper Functions

​convert_image_element

​draw_with_boxes

​table_elem_to_dataframe

​tables_to_pandas

Synchronous Partitioning Functions

partition_file

Aynchronous Partitioning Functions

partition_file_async_submit

partition_file_async_result

partition_file_async_cancel

partition_file_async_list

Helper Functions

convert_image_element

draw_with_boxes

table_elem_to_dataframe

tables_to_pandas