Adding tags to image

[Zain-Bin-Arshad]

I have done a lot using this package and I greatly appreciate your efforts. I want to add some alt text to an image. This text will be a MathML string so when any screen reader encounters an image it will check whether its has some MathML data and if yes then screen reader will read the mathematical equation.

Which API end-point should I use to achieve this. Thanks.

[JorjMcKie]

As you describe it, this is not supported in neither MuPDF nor in PyMuPDF.
Please provide an example PDF to see how that would look like - maybe there is an easy way using low-level code.

[JorjMcKie]

Interestingly, none of the main PDF specification manuals by Adobe mentions anything on mathematical markup language.

[JorjMcKie]

Please provide some feedback on this so I can follow-up or close this discussion item.

[Zain-Bin-Arshad]

I am just compiling some related resources, will update here in a bit .

[Zain-Bin-Arshad]

Sorry for the late response, @JorjMcKie if you download given PDF file you can see that it talks about IPDDomNode/IPDomElement methods that can be used to set the hidden attributes of an element (in my case image). Issue is its all in C I think and I don't want to go down on that road, I want to stick to PyMuPDF as always.
Once we can set the attribute then NVDA/MathPlayer can read the MathML version of the image and we are good to go.

Link: https://www.adobe.com/content/dam/acom/en/devnet/acrobat/pdfs/access.pdf

[Zain-Bin-Arshad]

Plus this SO link was also useful : https://stackoverflow.com/questions/51530305/adobe-acrobat-api-in-python

But again I can't find the documentation for Acrobat Python API, I think I am not looking in the right manner.

[JorjMcKie]

These references only talk about how to interface with Adobe Acrobat components. This has nothing to do with general PDF access and hence is unusable for PyMuPDF, too.
Also, IPDDomNode etc. are components of these APIs, and they only read information from PDFs (inside an environment set up by the Acrobat reader) - they do not set any properties (whether hidden or not) in PDF documents.
In the same sense, some Adobe Python API is of no relevance. I also looked at some of the MathML description. There is no mentioning on how and where this XML dialect (which it is) must be stored in PDF documents to achieve the effect you desire. This however is what I need to consider your request.
Please also remember, that PyMuPDF itself does not display documents - it is no PDF viewer like Adobe Acrobat, Nitro PDF, or evince or whatever. It stores information in PDFs in such a way, that can be interpreted by those viewer software.

So what I need you to tell me is something like: "Here is a string containing something. Put it in a PDF page / image / whatever ... object using key /Abracadabra ...".
I do not need to know what is contained in that string.
This is why I need an example PDF document, which works the way you like. Then we can decide on any implications.

[Zain-Bin-Arshad]

If you take a look at this PDF file, you can see that there are 3 images containing math equations. Example screenshot:

MathML string for these images:
Image 3.3: "\\( \\lim _{\\delta \\rightarrow 0}\\left[f(a)\\left(x_{1}-a\\right)+f\\left(x_{1}\\right)\\left(x_{2}-x_{1}\\right)+\\ldots+f\\left(x_{n-1}\\right)\\left(b-x_{n-1}\\right)\\right]=A \\)"

Image 3.4: "\\( \\lim _{\\delta \\rightarrow 0} \\sum_{i=1}^{n} f\\left(x_{i-1}\\right)\\left(x_{i}-x_{i-1}\\right)=A \\)"

Image 3.5: "\\( \\lim \\sum_{a}^{b} f(x) d x=A \\)"

So what I want is to save this MathML in a way that it gets associated with the image, so that when a screen reader like NVDA tries to read the image it can look for ok is there some MathML associated with this image, if there is then I can start reading it.

There must be a way like we have popup_annot that stores information which is not visible in the document but only when you interact with it an d it popups, I am missing out on something, I believe there is some straight forward way to achieve this.

Moreover, I want to THANK you for the effort you put in to help other. When I was creating pdf-viewer in python, your fast replies helped me a lot in adding features like night mode, magnification etc.

[JorjMcKie]

So what I want is to save this MathML in a way that it gets associated with the image, so that when a screen reader like NVDA tries to read the image it can look for ok is there some MathML associated with this image, if there is then I can start reading it.

I did understand stand already! Are you telling me, that somewhere in your PDF those XML data are hidden?

[Zain-Bin-Arshad]

No, data is not hidden anywhere in the document. I am using MathPix API to fetch the MathML version of each picture. So what I am planning to do is, I have a document, I will MathML of each image and store it in the PDF some how(this is where I need your help). Then I will save the new PDF with embedded MathML, when NVDA screen reader tries to read the image it will read the MathML and we are good.

[JorjMcKie]

store it in the PDF some how

We are back to start here: Where must these data be stored in a PDF?! Storing them just somehow won't help.
There already is plenty of PyMuPDF low-level API code available to store it in whatever way.

[Zain-Bin-Arshad]

Data must be stored in a way that it can be associated to an Image. Just like we can have metadata about an image right? So we can add another key to some kind of has which says something like alt or math_ml and then set the value of this key to the MathML string.

I don't know how can we achieve something like this, you are asking me to tell you 100% where to save it, even I don't know 100% I just have an idea. You know PyMuPDF 100 times more than me and sir you are in a much better place to suggest something here. I hope this information will help you. Thanks.

[JorjMcKie]

So we can add another key to some kind ...

Look, once again:
The question is not how to set a key and its value somewhere!

The questions are:

• What must be the name of that key?
• Which is the PDF object that must be given that key-value pair?

You are the one who must answer those questions.
I am going to tell you then how to store those data using PyMuPDF.
IF the PDF key is /MathML and IF the image object is the one who must be given those data and IF those data must be stored as a top-level key, then the following code will do the storing:

# xref is the image's cross reference number
# 'mathml' is the string containing the MathML source
# then:
doc.xref_set_key(xref, "MathML", fitz.getPDFstr(mathml))
# done

Function fitz.getPDFstr() take some unicode string and makes it PDF-compatible: putting brackets around it, escaping reserved PDF characters like backslashes and brackets and non-ASCII characters, etc.
The image xref can be found out by inspecting the list page.get_images(). If /MathML must be stored under some lower-level key of the image object - let us call it /Key for the moment - then the above statement must be modified to doc.xref_set_key(xref, "Key/MathML", fitz.getPDFstr(mathml)).

Several IFs which only you can clarify ...

[Zain-Bin-Arshad]

I am going to try this and I hope this will serve the purpose.

[JorjMcKie]

I can't image it will work.
It would have been documented otherwise and hence be detectable via Google.
But good luck!

[Zain-Bin-Arshad]

If you go here: https://www.webbie.org.uk/nvda/api/NVDAObjects.IAccessible.adobeAcrobat-pysrc.html#Document

You can see it makes use of IPDDomNode, IPDomElement just like the PDF that I mentioned in earlier comments(I think there is some relation between them) and if we can set the values there somehow in the PDF, we can make NVDA to read them. Well I am doing as much search as possible to fill in the dots. I am positive that there must be someway that needs to be dug out. Thanks

[niall-colgan]

I wonder has there been any progress on this? It's something I'm also interested in, specifically in relation to adding alt text for images in PDFs. I tried the following:

description = 'Test Alt for image'
for page_number in range(len(pdf_file)):
      page = pdf_file[page_number]
      image_list = page.get_images()
      for image_index, img in enumerate(image_list, start=1):
        xref = img[0]
        pdf_file.xref_set_key(xref, "Alt", f'({description})')
pdf_file.save('test.pdf')

but it doesn't seem to work as I was hoping. It does set the key but it doesn't show as alt text when you view the pdf inside Acrobat Reader or similar.

The PDF file attached is a sample file that contains alt tags on the images which show in Acrobat Reader, e.g.

accessible-pdf-example.pdf

Any suggestions as to whether this is possible with pymypdf?

[JorjMcKie]

It is not possible at all - not in the PDF specification. So it has nothing to do with PyMuPDF or any other tool.
All text in the vicinity of an image or vector graphic on a PDF page is just text. There is no PDF-relevant connection between these two things. As far as PDF is concerned, the closeness between an image and whatever text is pure chance.

You can add whatever key-value to any PDF object, for example /Alt (whatever text). This is fully legal. But it will be ignored by all PDF viewers because they don't know what to do with it.

[alireza-hariri]

The alt text for image (or other content) is part of the official spec of PDF 1.7 (ISO 32000)

It should be in the Logical Structure in the trailer (the /StructTreeRoot key).
See the section 14.7 (and 14.9.3) of the spec document.

[JorjMcKie]

This only applies to documents that have been tagged. This today still applies to a small minority of all PDFs out there.
If the PDF is however properly tagged, then even detection of tables and ordered / unordered lists is no issue any longer.